Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m1.737239956s trace: https://jaeger.dev.storj.io/trace/614e424f5910bbf9 FAIL (2025-12-08 17:44:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m1.737267496s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 498.521441ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:44:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 498.552657ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 442.610687ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:43:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.636167ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 490.592286ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:42:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 490.638025ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 479.143399ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:41:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.172196ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 659.868219ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:40:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 659.887594ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 560.79401ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:40:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 560.889444ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 568.896359ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:39:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 568.927178ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 527.998466ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:38:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.020861ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 477.116386ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:37:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.142009ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 501.131736ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:36:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.163625ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 499.099135ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.132157ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 844.228571ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:35:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 844.349076ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 412.872198ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:34:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.90194ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 547.643293ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:34:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 547.693208ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 479.963978ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:33:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.003151ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 450.102872ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:31:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.137626ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 481.613511ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:30:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 481.641904ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 501.611433ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:29:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.640707ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 472.144879ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:28:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.172107ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 444.311806ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:28:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.340415ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 478.088872ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:27:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.139235ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 438.318302ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:26:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 438.347477ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 511.64034ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:26:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.734513ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 427.95156ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:25:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.97463ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 519.922998ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:25:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 519.943706ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 442.486229ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:23:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.515569ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 431.729776ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:23:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.766542ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 582.536176ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:22:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 582.662752ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 407.526518ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:21:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.547563ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 475.968442ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:21:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.995012ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 632.262871ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:21:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 632.357087ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 536.940375ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:20:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.972909ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 459.415906ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:19:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.444388ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 421.229478ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:18:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.266026ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 497.701186ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:17:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.729426ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 480.92489ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:16:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.951632ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 468.591143ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:16:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 468.612442ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 524.951992ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:15:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.02007ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 461.805218ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:14:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.851554ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 459.637687ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:14:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.726207ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 523.647897ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:13:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 523.695657ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 449.118996ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:12:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.145569ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 446.683164ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:11:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.710479ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 498.219999ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:10:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 498.335502ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 579.403817ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 579.44046ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 427.233995ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:09:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.273717ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 478.918933ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:08:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.947408ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 469.843498ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.872001ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 502.585903ms trace: https://jaeger.dev.storj.io/trace/535d44beeeafe52a FAIL (2025-12-08 17:07:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 17:02:48.738023080 (73h34m49.565731479s) [upload=76480]: Starting upload Dec 8 17:02:48.738095838 (73h34m49.565804247s) [upload=76480]: Done starting upload Dec 8 17:02:48.738417931 (73h34m49.566126324s) [upload=76480]: Uploading segments... Dec 8 17:02:48.738470600 (73h34m49.566179003s) [upload=76480]: Got next segment. Inline: false Dec 8 17:02:48.738517579 (73h34m49.56622597s) [upload=76480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 17:02:48.738523576 (73h34m49.566231968s) [upload=76480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 17:02:48.769802661 (73h34m49.597511053s) [seg_pos={0 0}, upload=76480]: Begin upload segment... Dec 8 17:02:48.769865494 (73h34m49.597573886s) [seg_pos={0 0}, upload=76480]: Done begin upload segment. Dec 8 17:02:48.769907136 (73h34m49.597615538s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.769939359 (73h34m49.597647766s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770462858 (73h34m49.598171256s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770592023 (73h34m49.59830042s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770900372 (73h34m49.598608765s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.770964143 (73h34m49.598672541s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771340212 (73h34m49.599048606s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771369877 (73h34m49.599078273s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771701791 (73h34m49.599410175s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.771839972 (73h34m49.599548367s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772466849 (73h34m49.600175247s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.772890389 (73h34m49.600598785s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773397195 (73h34m49.601105589s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773517775 (73h34m49.601226156s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.773954120 (73h34m49.601662518s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.774310139 (73h34m49.602018536s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Uploading piece... Dec 8 17:02:48.799192405 (73h34m49.626900805s) [noise=true , address=77.90.31.13:4628 , piece_id=LTI5XLVJMTZ2U373UX2L4AMJEILTDR2PYFQBAF5DKEVNQSKD5CXA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.799876711 (73h34m49.627585112s) [noise=true , address=77.90.31.12:4589 , piece_id=LJUT5EHGQZJYA2AS5S53CJULTOK4LQYV3ZDZUJAFXLIY3BPO3SFQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.807505994 (73h34m49.635214394s) [noise=true , address=77.90.31.21:4877 , piece_id=3B56C74MQDSFAXO5SAZ2BG2QMHZZWSOX2BW3DACVRYW6F2QKDMQQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 17:02:48.845983839 (73h34m49.673692221s) [upload=76480]: Next returned nil segment Dec 8 17:02:48.845997556 (73h34m49.673705948s) [upload=76480]: Waiting for error group managing segments... Dec 8 17:02:48.853006781 (73h34m49.680715171s) [noise=true , address=145.79.227.6:4257 , piece_id=IKR5EZUXWCEVA5SQ7546KXSKWTBDYQJ7AMTVXX2T25IOOMW2TKCQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.853644007 (73h34m49.681352402s) [noise=true , address=145.79.227.12:4765 , piece_id=R6SQ4SBMCWD4DKP4FAGLPUMSOA4KRF4F2AML2TV464ZYIQ537JFA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856107946 (73h34m49.68381634s) [noise=true , address=145.79.227.22:4510 , piece_id=KVY33OABWKC4AHQXGQ2QQZBB6TNGSSSR4ZOZN56NHBGGBJUQEYLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856510829 (73h34m49.684219222s) [noise=true , address=145.79.227.26:4654 , piece_id=2FHPIZA5BL7Q4JQQZE56KLIUKCNUW7CUARG72SP4Q7YBLEV7VA7A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856772442 (73h34m49.684480837s) [noise=true , address=145.79.227.34:4907 , piece_id=YGIATRQPO6ZWFZD45CSFBSVL7L3DRNGIC5RJGPPOV6CSNXDNEDLQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.856975680 (73h34m49.684684071s) [noise=true , address=145.79.227.29:4941 , piece_id=P2Z5FPJDTTTKH3ZL4UU3JW6NC2VJ34RG4CIMAXRW6Z5EZPHUQQ6Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857242047 (73h34m49.68495045s) [noise=true , address=145.79.227.24:4583 , piece_id=VWEGXRGYJXX7H75WUTBDZFZ3HPR74VPII4K6SJPBFV62FCXS5JIQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857377588 (73h34m49.68508598s) [noise=true , address=145.79.227.28:4726 , piece_id=SBWY4VB3BN65MOVJO22NIR2HN7CS63ODABH4XH3O55NDHCCMCM2A, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857752465 (73h34m49.68546086s) [noise=true , address=145.79.227.23:4547 , piece_id=FKELFFQLGRPRX5L7FKMDZPOTBDHMKQDP6HJQUYLUDVMXOSKIMXGA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.857759093 (73h34m49.685467489s) [noise=true , address=145.79.227.23:4546 , piece_id=L2UX5LJADQOQVS7ZKHBOTHKZNXXV22W4UJ3OUGCSHWUDMHT4EYRA, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.858219382 (73h34m49.685927774s) [noise=true , address=145.79.227.27:4690 , piece_id=IWPD3QZEMPXRZVSDS7S6UEIHELJEVVWIECBE527FNIF25ZG2NP2Q, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:02:48.859176521 (73h34m49.686884917s) [noise=true , address=145.79.227.41:4086 , piece_id=QVKEINCWGAGSEBYCFWPHLGVDRLQLVT35LZCCCYF3VSKTWLY5EYRQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: <nil> Dec 8 17:03:18.698437167 (73h35m19.526145561s) [noise=true , address=77.90.31.13:4627 , piece_id=DCNTHYBRJN5Q2IWGPKRQ7LEKYCSXOQSLQ5GI4YGPNGVQINLQKIYQ, seg_pos={0 0}, upload=76480]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: expected piece hash; context deadline exceeded; EOF Dec 8 17:03:18.698506535 (73h35m19.526214917s) [seg_pos={0 0}, upload=76480]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 17:03:18.724186991 (73h35m19.551895387s) [upload=76480]: Done uploading segments... Dec 8 17:03:18.724296376 (73h35m19.552004781s) [upload=76480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 502.623167ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)