Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 148722 reports (Page 71 of 2975)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X.......X.....................................X........] duration: 536.401945ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 22:00:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 536.439306ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.............................................X........] duration: 566.695492ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:00:25 UTC) orbiter warning: unexpected EOF Duration: 566.747728ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......X......................................X.......] duration: 496.541743ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 22:00:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.600633ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.............................................X.......] duration: 679.288416ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 22:00:06 UTC) orbiter warning: unexpected EOF Duration: 679.416804ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X......................................X........] duration: 723.273991ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 22:00:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 723.307613ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.................................X.........] duration: 515.056432ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:59:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 515.081809ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.......X......................................X......] duration: 615.996025ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 21:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 616.023347ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.....X.........................................] duration: 442.008652ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:59:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 442.044303ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............................................X......] duration: 1.009880393s trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 21:59:41 UTC) orbiter warning: unexpected EOF Duration: 1.009906622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X.......X.....................................X.......] duration: 531.865571ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:59:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 531.913832ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.................................X........] duration: 878.773257ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:59:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 878.798861ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......................................X.......] duration: 609.363701ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:59:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 609.392413ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.....X........................................] duration: 443.712224ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:59:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 443.73635ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......................................X......] duration: 500.110716ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:59:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 500.123599ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.................................X.......] duration: 476.302574ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:58:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.328662ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....X.......................................] duration: 428.695408ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:58:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 428.793722ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......X......................................X.....] duration: 467.980963ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 21:58:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.994497ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X.............................................X.....] duration: 641.852325ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 21:58:54 UTC) orbiter warning: unexpected EOF Duration: 641.883236ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.....X......................................] duration: 511.487481ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:58:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 511.517269ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X.......X.....................................X......] duration: 754.154113ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:58:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 754.183385ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............................................X....] duration: 636.022526ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 21:58:28 UTC) orbiter warning: unexpected EOF Duration: 636.114668ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X.......X.....................................X.....] duration: 464.903701ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:58:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.924931ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X......................................X.....] duration: 524.369131ms trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:58:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 524.397797ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................................X......] duration: 516.359589ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:57:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 516.401159ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.......X......................................X....] duration: 458.880328ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 21:57:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.903626ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X.............................................X...] duration: 589.141487ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 21:57:36 UTC) orbiter warning: unexpected EOF Duration: 589.169443ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................................X.....] duration: 483.269158ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:57:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 483.368612ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.....X.....................................] duration: 1.31226157s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:57:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.312294002s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X......................................X....] duration: 1.345421171s trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:57:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.34545643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................................X....] duration: 508.629269ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:57:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 508.684699ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......X......................................X...] duration: 504.914501ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 21:57:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.94852ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X.......X.....................................X....] duration: 685.04722ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:56:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 685.077001ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X......................................X...] duration: 7.205937414s trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:56:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.205965728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X.......X.....................................X...] duration: 571.327289ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:56:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 571.353319ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X..........X.............................................X..] duration: 721.749855ms trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 21:56:36 UTC) unexpected EOF Duration: 721.776066ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.....X....................................] duration: 430.113061ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:56:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 430.141088ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......X......................................X..] duration: 500.855757ms trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 21:56:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 500.884093ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....................X.................................X...] duration: 456.388244ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:56:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.417422ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X..........X.............................................X.] duration: 1.40531052s trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 21:55:59 UTC) unexpected EOF Duration: 1.405324497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X..........X.............................................X] duration: 30.303343918s trace: https://jaeger.dev.storj.io/trace/25b2c21daa3d085 FAIL (2025-10-31 21:55:57 UTC) unexpected EOF Duration: 30.303361694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.................X......................................X..] duration: 2.080374267s trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:55:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.080403483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....X...................................] duration: 1.5842361s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:55:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1.584281218s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X.......X.....................................X..] duration: 730.385011ms trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:55:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 730.412085ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....................X.................................X..] duration: 896.496193ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:55:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 896.521388ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X..........X.............................................] duration: 53.534784203s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:55:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.534820863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.....X..................................] duration: 4.069162884s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:55:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 4.069187538s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......X......................................X.] duration: 1.044290361s trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 21:55:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.04431745s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.................X......................................X.] duration: 9.911194339s trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:55:05 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 9.911214593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X.......X.....................................X.] duration: 1.182639865s trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:54:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.182662934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.................X......................................X] duration: 1m0.023791193s trace: https://jaeger.dev.storj.io/trace/41c2b066eb9ea5e8 FAIL (2025-10-31 21:54:55 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.243590254 (34h14m25.907539233s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244078634 (34h14m25.908027607s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.244974253 (34h14m25.908923236s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.245239075 (34h14m25.909188054s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246191747 (34h14m25.910140732s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.246721069 (34h14m25.910670051s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247101943 (34h14m25.911050925s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247367511 (34h14m25.911316501s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247624227 (34h14m25.911573229s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.247887850 (34h14m25.911836838s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248392598 (34h14m25.912341583s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.248921469 (34h14m25.912870457s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249532021 (34h14m25.913481008s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.249926757 (34h14m25.913875738s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250408522 (34h14m25.91435751s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.250881171 (34h14m25.914830153s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.251325874 (34h14m25.915274858s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Uploading piece... Oct 31 21:53:55.319899795 (34h14m25.983848763s) [upload=35594]: Next returned nil segment Oct 31 21:53:55.319912385 (34h14m25.983861365s) [upload=35594]: Waiting for error group managing segments... Oct 31 21:53:55.332186509 (34h14m25.996135498s) [noise=true , address=145.79.227.3:4181 , piece_id=H6V4HL6HBTYZZX2CU42A7N2GZW5V3VVNUH2U7DJGVTDSMIJT5ETA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.337196032 (34h14m26.001145012s) [noise=true , address=145.79.227.14:4852 , piece_id=SOPXC7PIR66BWUKRNJHNGG6NN2QAYQ4LSOWISV6VPEH65YECBOHA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.345822268 (34h14m26.009771248s) [noise=true , address=145.79.227.47:4275 , piece_id=NVD667NEYMIMVWHOIIQEBZENEGR5NMMFER6PYX4PBG3SXSRJOGFA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.356624949 (34h14m26.020573918s) [noise=true , address=145.79.227.44:4228 , piece_id=J6XJCH3UF7IFRNHEKYPZ2VQWHG6FZNX2TNQP3Y4KYWX66XYYK4VA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.357992611 (34h14m26.021941606s) [noise=true , address=145.79.227.13:4818 , piece_id=WIFPCYIMOLHRMVFM6ZARXCE7VH6GIBQ2Q3TVGWGL6BYOQK4HHMUQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.358284965 (34h14m26.022233957s) [noise=true , address=145.79.227.19:4090 , piece_id=MFVZOHBOGGEG7ND4PX6SA6AWIWNCR43LJBPL327TD6ULAEU5HJJA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.363582356 (34h14m26.027531346s) [noise=true , address=145.79.227.36:4536 , piece_id=2MKOMZYMYP3R5ODU6NTATHLGJLOJ2LEVTOU7YMTCK3M3LM2TZMQA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.376206640 (34h14m26.040155641s) [noise=true , address=145.79.227.6:4274 , piece_id=VKYLWRZMJ5QHXYCDDE7FVHUKVXFWLSTMM6WEDL6VIWXDFBBX72OA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:53:55.386414599 (34h14m26.050363579s) [noise=true , address=145.79.227.52:4465 , piece_id=BPMYKQI3N526E5IY7DJK64ZKB4XHYDEEFB5QV5SEFRSOHBGJ3AJQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.030428990 (34h15m20.694377969s) [noise=true , address=45.91.116.12:4235 , piece_id=GUST4ADMDZGJIC5K5BOKXYTPTEBGYGDXMGXICQI3SRGZIMVNSNOA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:50.506293399 (34h15m21.170242379s) [noise=true , address=45.91.116.10:4103 , piece_id=ASJ3TFEWF7FVBBCHCJJ46UUZRZVMQTPMQ4KTOSX7LCFZ3WZUGCZQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: <nil> Oct 31 21:54:55.184164260 (34h15m25.848113246s) [noise=true , address=45.91.116.14:4203 , piece_id=DXLZT7TNAP7BZXQU535QHNTE36U6IR2TMZQ3RQTBXNVRQMOGGXDQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33840->45.91.116.14:4203: use of closed network connection Oct 31 21:54:55.184248337 (34h15m25.848197346s) [noise=true , address=77.90.31.13:4641 , piece_id=TRVZJ5V45LZAZC6QIDEEO4BENKEVOSUSV4Q4T2TN6GXDVORVJR3Q, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184319241 (34h15m25.84826822s) [noise=true , address=77.90.31.11:4560 , piece_id=5SDNYH5KNI7YFLHMPMQKWF67VC23THJDZ4UH3ASCKK2JZJ26ZWIA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47386->77.90.31.11:4560: use of closed network connection Oct 31 21:54:55.184492039 (34h15m25.848441022s) [noise=true , address=77.90.31.19:4861 , piece_id=WPOE42O4X7RAOCEODWDEWBLCSWREVQHHTJYVGF6LQJDL3XXI3R7A, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:55.184662449 (34h15m25.848611434s) [noise=true , address=45.91.116.4:4964 , piece_id=SJGH4Q3GXKC2YKNR3KPKZEPWD5SJJGSQLUCDIIWBLGZPEQOIY2DQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: read tcp 159.223.143.39:38358->45.91.116.4:4964: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184739659 (34h15m25.848688638s) [noise=true , address=77.90.31.14:4686 , piece_id=DAL3FB24LF3FHXWBB3R5XITLPAOSROJYVBE7BIJQ676HVL4P7QPA, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: write tcp 159.223.143.39:41986->77.90.31.14:4686: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184756157 (34h15m25.848705142s) [noise=true , address=45.91.116.13:4177 , piece_id=C6RUPGPDPO2VRTOXGQM7VHYZEDATQX5IZBWA73I5QY4LFHP6R6MQ, seg_pos={0 0}, upload=35594]: Done uploading piece. err: ecclient: upload failed (node:1xwaFojWAo54xD34ZA7HACYmJf811KFqN9biJJoag5oxE9fJqx, address:45.91.116.13:4177): protocol: read tcp 159.223.143.39:41676->45.91.116.13:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:55.184786251 (34h15m25.848735222s) [seg_pos={0 0}, upload=35594]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Oct 31 21:54:55.206244634 (34h15m25.870193618s) [upload=35594]: Done uploading segments... Oct 31 21:54:55.206282843 (34h15m25.870231833s) [upload=35594]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.023821903s core uplink us-select-1 us1-placement-20

1
...
69
70
71
72
73
...
2975