Orbiter History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 398.659522ms trace: https://jaeger.dev.storj.io/trace/348544fb101a1fec FAIL (2026-03-10 14:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:12:46.924474189 (99h12m44.521245216s) [upload=111047]: Starting upload Mar 10 14:12:46.924609401 (99h12m44.521380427s) [upload=111047]: Done starting upload Mar 10 14:12:46.924713909 (99h12m44.521484924s) [upload=111047]: Uploading segments... Mar 10 14:12:46.924924341 (99h12m44.521695367s) [upload=111047]: Got next segment. Inline: false Mar 10 14:12:46.924976798 (99h12m44.52174782s) [upload=111047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:12:46.924983073 (99h12m44.521754096s) [upload=111047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:12:47.071618567 (99h12m44.668389592s) [upload=111047]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:12:47.071640064 (99h12m44.668411079s) [upload=111047]: Done uploading segments... Mar 10 14:12:47.071683313 (99h12m44.668454336s) [upload=111047]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 398.753543ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 409.667765ms trace: https://jaeger.dev.storj.io/trace/c2d8083a6acdc86 FAIL (2026-03-10 14:12:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:12:00.685009330 (99h11m58.281780343s) [upload=111036]: Starting upload Mar 10 14:12:00.685079460 (99h11m58.281850485s) [upload=111036]: Done starting upload Mar 10 14:12:00.685297072 (99h11m58.282068089s) [upload=111036]: Uploading segments... Mar 10 14:12:00.685608852 (99h11m58.282379877s) [upload=111036]: Got next segment. Inline: false Mar 10 14:12:00.685670596 (99h11m58.282441623s) [upload=111036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:12:00.685676790 (99h11m58.282447813s) [upload=111036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:12:00.852821672 (99h11m58.449592685s) [upload=111036]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:12:00.852890838 (99h11m58.449661869s) [upload=111036]: Done uploading segments... Mar 10 14:12:00.852974868 (99h11m58.449745884s) [upload=111036]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 409.748542ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 361.25109ms trace: https://jaeger.dev.storj.io/trace/1e1d5a6e82f5bf29 FAIL (2026-03-10 14:11:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:11:11.294065127 (99h11m8.890836151s) [upload=111020]: Starting upload Mar 10 14:11:11.294225375 (99h11m8.890996393s) [upload=111020]: Done starting upload Mar 10 14:11:11.294357907 (99h11m8.89112893s) [upload=111020]: Uploading segments... Mar 10 14:11:11.294816239 (99h11m8.891587262s) [upload=111020]: Got next segment. Inline: false Mar 10 14:11:11.294899738 (99h11m8.891670761s) [upload=111020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:11:11.294906921 (99h11m8.891677944s) [upload=111020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:11:11.424421665 (99h11m9.021192697s) [upload=111020]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:11:11.424755991 (99h11m9.021527018s) [upload=111020]: Done uploading segments... Mar 10 14:11:11.424859428 (99h11m9.021630459s) [upload=111020]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 361.276012ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 379.103749ms trace: https://jaeger.dev.storj.io/trace/2a7f01a4a32b690e FAIL (2026-03-10 14:10:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:10:32.712695488 (99h10m30.309466514s) [upload=111008]: Starting upload Mar 10 14:10:32.712749975 (99h10m30.309521008s) [upload=111008]: Done starting upload Mar 10 14:10:32.712897244 (99h10m30.309668275s) [upload=111008]: Uploading segments... Mar 10 14:10:32.713157967 (99h10m30.309928991s) [upload=111008]: Got next segment. Inline: false Mar 10 14:10:32.713231959 (99h10m30.310002992s) [upload=111008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:10:32.713239511 (99h10m30.310010538s) [upload=111008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:10:32.832186009 (99h10m30.428957039s) [upload=111008]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:10:32.832208633 (99h10m30.428979661s) [upload=111008]: Done uploading segments... Mar 10 14:10:32.832308741 (99h10m30.429079765s) [upload=111008]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 379.126484ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 411.702246ms trace: https://jaeger.dev.storj.io/trace/42bc5cda72693c31 FAIL (2026-03-10 14:09:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:09:31.575761145 (99h9m29.172532172s) [upload=110992]: Starting upload Mar 10 14:09:31.575821971 (99h9m29.172593004s) [upload=110992]: Done starting upload Mar 10 14:09:31.576111562 (99h9m29.172882577s) [upload=110992]: Uploading segments... Mar 10 14:09:31.576190116 (99h9m29.17296115s) [upload=110992]: Got next segment. Inline: false Mar 10 14:09:31.576241365 (99h9m29.173012387s) [upload=110992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:09:31.576247081 (99h9m29.173018105s) [upload=110992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:09:31.718066019 (99h9m29.314837045s) [upload=110992]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:09:31.718087630 (99h9m29.314858657s) [upload=110992]: Done uploading segments... Mar 10 14:09:31.718120728 (99h9m29.314891739s) [upload=110992]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 411.724069ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 386.423696ms trace: https://jaeger.dev.storj.io/trace/39c08ce75bcace6b FAIL (2026-03-10 14:08:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:08:37.743557342 (99h8m35.340328356s) [upload=110977]: Starting upload Mar 10 14:08:37.743624415 (99h8m35.340395442s) [upload=110977]: Done starting upload Mar 10 14:08:37.743717545 (99h8m35.340488569s) [upload=110977]: Uploading segments... Mar 10 14:08:37.744182662 (99h8m35.340953694s) [upload=110977]: Got next segment. Inline: false Mar 10 14:08:37.744285254 (99h8m35.341056288s) [upload=110977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:08:37.744293595 (99h8m35.341064628s) [upload=110977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:08:37.870508466 (99h8m35.467279496s) [upload=110977]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:08:37.870529375 (99h8m35.467300401s) [upload=110977]: Done uploading segments... Mar 10 14:08:37.870572010 (99h8m35.467343027s) [upload=110977]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 386.492992ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 408.194284ms trace: https://jaeger.dev.storj.io/trace/7ac60f7a137a3aef FAIL (2026-03-10 14:07:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:07:40.849910045 (99h7m38.446681075s) [upload=110957]: Starting upload Mar 10 14:07:40.849991993 (99h7m38.446763009s) [upload=110957]: Done starting upload Mar 10 14:07:40.850415586 (99h7m38.447186606s) [upload=110957]: Uploading segments... Mar 10 14:07:40.850602684 (99h7m38.447373708s) [upload=110957]: Got next segment. Inline: false Mar 10 14:07:40.850750446 (99h7m38.447521491s) [upload=110957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:07:40.850758577 (99h7m38.447529604s) [upload=110957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:07:41.009763493 (99h7m38.606534543s) [upload=110957]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:07:41.009785655 (99h7m38.606556684s) [upload=110957]: Done uploading segments... Mar 10 14:07:41.009835875 (99h7m38.606606887s) [upload=110957]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 408.212293ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 469.939973ms trace: https://jaeger.dev.storj.io/trace/4f32ebc5248508ba FAIL (2026-03-10 14:06:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 10 14:06:45.730658599 (99h6m43.327429625s) [upload=110942]: Starting upload Mar 10 14:06:45.730766277 (99h6m43.327537316s) [upload=110942]: Done starting upload Mar 10 14:06:45.731178890 (99h6m43.327949899s) [upload=110942]: Uploading segments... Mar 10 14:06:45.731275423 (99h6m43.328046446s) [upload=110942]: Got next segment. Inline: false Mar 10 14:06:45.731327132 (99h6m43.328098163s) [upload=110942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 14:06:45.731334444 (99h6m43.328105468s) [upload=110942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 14:06:45.942780872 (99h6m43.539551899s) [upload=110942]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 10 14:06:45.942800844 (99h6m43.539571868s) [upload=110942]: Done uploading segments... Mar 10 14:06:45.942887285 (99h6m43.539658297s) [upload=110942]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Duration: 469.966041ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 405.352526ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:15:25 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 405.379429ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 434.400054ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:14:58 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 434.425226ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 408.990912ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:14:10 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 409.041467ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 441.780126ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:13:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 441.861157ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 396.702394ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:12:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 396.728177ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 391.986754ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:11:33 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 392.007923ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 409.492078ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:10:40 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 409.547056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 546.538843ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:10:03 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 546.577349ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 513.146988ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:08:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 513.169989ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 471.610074ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:07:59 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 471.628521ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 533.354898ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:07:02 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 533.393955ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 496.635767ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:05:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 496.720567ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 407.106795ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:05:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 407.142511ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 450.514656ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:05:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 450.564749ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 405.702206ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:04:30 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 405.728888ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 434.876324ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:03:59 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 434.90827ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 454.754544ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:03:18 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 454.774894ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 422.385014ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:02:30 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 422.400233ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 430.543147ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:02:06 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 430.565963ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 476.373103ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:01:43 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 476.421251ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 460.116467ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:00:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 460.140877ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 540.813597ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 04:00:23 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 540.845102ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 390.329173ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:59:59 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 390.36149ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 495.745257ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:58:58 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 495.771054ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 385.456261ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:58:13 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 385.489455ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 439.093157ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:57:38 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 439.119805ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 405.406372ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:56:50 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 405.432272ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 411.895131ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:55:43 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 411.922635ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 424.889051ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:55:09 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 424.917827ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 414.094733ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:54:14 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 414.128927ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 443.075065ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:53:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 443.094639ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 401.908387ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:53:22 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 401.984595ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 418.156528ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:52:50 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 418.185537ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 418.29795ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:51:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 418.327652ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 469.216511ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:50:37 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 469.257408ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 450.505279ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:49:37 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 450.528547ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 392.24467ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:48:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 392.274075ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 419.737491ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:47:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 419.767506ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 381.829715ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:47:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 381.856953ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 519.684035ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:46:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 519.713807ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 412.835227ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:45:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 412.863891ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 391.454772ms trace: https://jaeger.dev.storj.io/trace/15d9699ef5856439 FAIL (2026-03-10 03:44:35 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context canceled ======LOGS====== Mar 10 03:30:26.400470515 (88h30m23.99724153s) [upload=99029]: Starting upload Mar 10 03:30:26.400549920 (88h30m23.997320949s) [upload=99029]: Done starting upload Mar 10 03:30:26.400723479 (88h30m23.997494492s) [upload=99029]: Uploading segments... Mar 10 03:30:26.400916202 (88h30m23.99768723s) [upload=99029]: Got next segment. Inline: false Mar 10 03:30:26.401010555 (88h30m23.997781583s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 03:30:26.401018253 (88h30m23.997789279s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 03:30:26.430477746 (88h30m24.027248772s) [seg_pos={0 0}, upload=99029]: Begin upload segment... Mar 10 03:30:26.430560181 (88h30m24.027331193s) [seg_pos={0 0}, upload=99029]: Done begin upload segment. Mar 10 03:30:26.430594758 (88h30m24.027365792s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.430627619 (88h30m24.027398651s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.431330226 (88h30m24.02810125s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432084089 (88h30m24.028855104s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432773019 (88h30m24.029544035s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.432917989 (88h30m24.02968902s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.434283438 (88h30m24.031054471s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.436329199 (88h30m24.033100222s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.443120493 (88h30m24.039891522s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.445208438 (88h30m24.041979471s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.448571281 (88h30m24.045342309s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450175052 (88h30m24.046946085s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450435387 (88h30m24.047206416s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.450749876 (88h30m24.04752089s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.451942610 (88h30m24.048713638s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.457706199 (88h30m24.054477232s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Uploading piece... Mar 10 03:30:26.537289064 (88h30m24.134060074s) [upload=99029]: Next returned nil segment Mar 10 03:30:26.537309578 (88h30m24.134080613s) [upload=99029]: Waiting for error group managing segments... Mar 10 03:30:26.543428198 (88h30m24.140199222s) [noise=true , address=45.91.116.9:4295 , piece_id=7KBW3P7HDBHB4MAQ3W3DH2STVAAST3KFVT47H6TN4FEKDZ6CHE6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.543509654 (88h30m24.140280688s) [noise=true , address=45.91.116.11:4121 , piece_id=QAZKSEQMVNWJEZFD23GKR754K5A7BEAHBLNZWJ7Y6LGV57U5SARA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.545528352 (88h30m24.142299363s) [noise=true , address=145.79.227.19:4082 , piece_id=GW3627XI3IW2UDAY2CDLEZUSKR5TB75LPYMJD3QEGU6AMIB5OWPA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.546079396 (88h30m24.142850422s) [noise=true , address=145.79.227.47:4265 , piece_id=A5AU3QAJY2SB6SHRHQKJ6NZZ3VE3472JSU3NQAZRK7YD5PNMW5JQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547067551 (88h30m24.143838563s) [noise=true , address=145.79.227.15:4873 , piece_id=OB7T2XEXSNVHXRN2GKS7FH2OB3XBR3XAZBORMMNJOHQPZ3XQQHKA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.547475238 (88h30m24.144246251s) [noise=true , address=79.127.219.41:4928 , piece_id=LAYU66NEWQNMDNMSI5UDJKC43SPAGLD2WGOILURPOLBFLGHIBMDA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.548670515 (88h30m24.14544153s) [noise=true , address=145.79.227.38:4980 , piece_id=BKU6KJJAQJEXGOMCHILD5QFYZQVCR62ZWXIAFIZTQNUPSAK42PZA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.549072511 (88h30m24.145843524s) [noise=true , address=145.79.227.18:4011 , piece_id=X7DJEF5HXNOXGE3ESESXFJXFIGDFGX4YJLHWM2E3U6T3XATHQJ7A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555074806 (88h30m24.151845822s) [noise=true , address=145.79.227.40:4052 , piece_id=NEQXES4FHKDBSDM54YVOEN4BDKITGSATSNU3OTUMODTJIPCCFHGA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555202619 (88h30m24.151973632s) [noise=true , address=145.79.227.8:4332 , piece_id=HFWYYBL54ELK3JCDVOTFD3E4YKXUPYJ6TELZFLYGZIY6KMIP4S6A, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.555891112 (88h30m24.152662122s) [noise=true , address=145.79.227.3:4153 , piece_id=QGHPYR3BBFESMADBC4YTROTTJSDFHDDBK3XPK2NSKFMRXTZSJDEA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.556216023 (88h30m24.152987036s) [noise=true , address=145.79.227.33:4875 , piece_id=DCZIABX6B3KSZVVWHXQMJQ76ILIMWDUU3UHWFI2M4XR2HE3ACA3Q, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560025999 (88h30m24.156797026s) [noise=true , address=145.79.227.39:4017 , piece_id=OSBLSGKAPIISJJBO7ET3P65Y3Z2RFNR3BLZCHZVXFHOHEO3OEDJA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: <nil> Mar 10 03:30:26.560054868 (88h30m24.156825905s) [seg_pos={0 0}, upload=99029]: Satisfied cohort requirements after 13 pieces Mar 10 03:30:26.560366952 (88h30m24.157137987s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560380635 (88h30m24.157151672s) [noise=true , address=145.79.227.50:4375 , piece_id=J2D5BZZMHU6BEQI3OQWWN6CCKAC4WVPLAIXR5U77C6QXSRHMFKYQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560411581 (88h30m24.157182632s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB): protocol: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34400->145.79.227.48:4305: use of closed network connection Mar 10 03:30:26.560424741 (88h30m24.157195767s) [noise=true , address=145.79.227.48:4305 , piece_id=UU37UIOZJJPIURNP5I4MP6NUL43U42ULQ2GXLDX6N53XHZCUI5WQ, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560869502 (88h30m24.157640525s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Mar 10 03:30:26.560886191 (88h30m24.157657204s) [noise=true , address=145.79.227.7:4297 , piece_id=QOAMRDHQLI62XVZQF2OH4SF2WZX2OZPULWSIRV5OEIYSARTVG3YA, seg_pos={0 0}, upload=99029]: Long tail cancellation due to optimal threshold Mar 10 03:30:26.560939806 (88h30m24.157710883s) [seg_pos={0 0}, upload=99029]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 10 03:30:26.561043132 (88h30m24.157814155s) [upload=99029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 03:30:26.561049781 (88h30m24.157820812s) [upload=99029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 03:30:26.608097299 (88h30m24.20486833s) [upload=99029]: Done uploading segments... Mar 10 03:30:26.608117724 (88h30m24.204888753s) [upload=99029]: Upload finished. err: <nil> Duration: 391.48318ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)