Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 1.302139643s trace: https://jaeger.dev.storj.io/trace/277ba948f183dc4f FAIL (2026-03-10 18:46:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:44:18.395709628 (103h44m15.992480658s) [upload=116117]: Starting upload Mar 10 18:44:18.395810068 (103h44m15.992581086s) [upload=116117]: Done starting upload Mar 10 18:44:18.395929782 (103h44m15.992700811s) [upload=116117]: Uploading segments... Mar 10 18:44:18.396421871 (103h44m15.993192902s) [upload=116117]: Got next segment. Inline: false Mar 10 18:44:18.396541809 (103h44m15.993312834s) [upload=116117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:44:18.396549171 (103h44m15.993320203s) [upload=116117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:44:18.428758532 (103h44m16.025529561s) [seg_pos={0 0}, upload=116117]: Begin upload segment... Mar 10 18:44:18.428822285 (103h44m16.0255933s) [seg_pos={0 0}, upload=116117]: Done begin upload segment. Mar 10 18:44:18.428861616 (103h44m16.02563264s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.429059881 (103h44m16.025830906s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430270915 (103h44m16.027041946s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430727660 (103h44m16.027498692s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431207878 (103h44m16.027978905s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431250249 (103h44m16.028021271s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431644829 (103h44m16.028415855s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431798421 (103h44m16.028569448s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432077244 (103h44m16.028848275s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432428854 (103h44m16.029199881s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432499385 (103h44m16.029270419s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432975554 (103h44m16.029746589s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.433599251 (103h44m16.030370283s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434278354 (103h44m16.031049379s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434667011 (103h44m16.031438044s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435223698 (103h44m16.031994733s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435757687 (103h44m16.032528714s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.436241820 (103h44m16.033012853s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.512361085 (103h44m16.109132112s) [upload=116117]: Next returned nil segment Mar 10 18:44:18.512381685 (103h44m16.109152703s) [upload=116117]: Waiting for error group managing segments... Mar 10 18:44:18.516528033 (103h44m16.113299062s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.518829594 (103h44m16.115600608s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.519605999 (103h44m16.116377021s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533489430 (103h44m16.130260444s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533564325 (103h44m16.130335338s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.774929408 (103h44m16.371700437s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.792542126 (103h44m16.389313153s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.811444852 (103h44m16.408215877s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.009790186 (103h44m16.606561214s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.017479102 (103h44m16.614250127s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.032351604 (103h44m16.629122617s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.036028441 (103h44m16.632799471s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.038165250 (103h44m16.634936273s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:45:18.350922849 (103h45m15.947693883s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: write tcp 159.223.143.39:49376->77.90.31.13:4638: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351159799 (103h45m15.947930832s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 159.223.143.39:42258->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351372791 (103h45m15.948143825s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:46988->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351533653 (103h45m15.948304686s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 159.223.143.39:49664->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.352230657 (103h45m15.949001693s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:45:18.352270159 (103h45m15.949041186s) [seg_pos={0 0}, upload=116117]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:45:18.373031667 (103h45m15.9698027s) [upload=116117]: Done uploading segments... Mar 10 18:45:18.373152165 (103h45m15.969923198s) [upload=116117]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.302161884s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.......X.....] duration: 1.30566447s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:46:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.305682329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..X..............................] duration: 11.777864555s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:46:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 11.777891931s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..................X...] duration: 630.372629ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:46:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 630.432081ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..X.............................] duration: 14.371757159s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:46:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 14.371818108s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 582.254255ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:46:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.292009ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 1.292340774s trace: https://jaeger.dev.storj.io/trace/277ba948f183dc4f FAIL (2026-03-10 18:45:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:44:18.395709628 (103h44m15.992480658s) [upload=116117]: Starting upload Mar 10 18:44:18.395810068 (103h44m15.992581086s) [upload=116117]: Done starting upload Mar 10 18:44:18.395929782 (103h44m15.992700811s) [upload=116117]: Uploading segments... Mar 10 18:44:18.396421871 (103h44m15.993192902s) [upload=116117]: Got next segment. Inline: false Mar 10 18:44:18.396541809 (103h44m15.993312834s) [upload=116117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:44:18.396549171 (103h44m15.993320203s) [upload=116117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:44:18.428758532 (103h44m16.025529561s) [seg_pos={0 0}, upload=116117]: Begin upload segment... Mar 10 18:44:18.428822285 (103h44m16.0255933s) [seg_pos={0 0}, upload=116117]: Done begin upload segment. Mar 10 18:44:18.428861616 (103h44m16.02563264s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.429059881 (103h44m16.025830906s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430270915 (103h44m16.027041946s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430727660 (103h44m16.027498692s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431207878 (103h44m16.027978905s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431250249 (103h44m16.028021271s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431644829 (103h44m16.028415855s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431798421 (103h44m16.028569448s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432077244 (103h44m16.028848275s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432428854 (103h44m16.029199881s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432499385 (103h44m16.029270419s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432975554 (103h44m16.029746589s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.433599251 (103h44m16.030370283s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434278354 (103h44m16.031049379s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434667011 (103h44m16.031438044s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435223698 (103h44m16.031994733s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435757687 (103h44m16.032528714s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.436241820 (103h44m16.033012853s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.512361085 (103h44m16.109132112s) [upload=116117]: Next returned nil segment Mar 10 18:44:18.512381685 (103h44m16.109152703s) [upload=116117]: Waiting for error group managing segments... Mar 10 18:44:18.516528033 (103h44m16.113299062s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.518829594 (103h44m16.115600608s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.519605999 (103h44m16.116377021s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533489430 (103h44m16.130260444s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533564325 (103h44m16.130335338s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.774929408 (103h44m16.371700437s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.792542126 (103h44m16.389313153s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.811444852 (103h44m16.408215877s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.009790186 (103h44m16.606561214s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.017479102 (103h44m16.614250127s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.032351604 (103h44m16.629122617s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.036028441 (103h44m16.632799471s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.038165250 (103h44m16.634936273s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:45:18.350922849 (103h45m15.947693883s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: write tcp 159.223.143.39:49376->77.90.31.13:4638: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351159799 (103h45m15.947930832s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 159.223.143.39:42258->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351372791 (103h45m15.948143825s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:46988->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351533653 (103h45m15.948304686s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 159.223.143.39:49664->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.352230657 (103h45m15.949001693s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:45:18.352270159 (103h45m15.949041186s) [seg_pos={0 0}, upload=116117]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:45:18.373031667 (103h45m15.9698027s) [upload=116117]: Done uploading segments... Mar 10 18:45:18.373152165 (103h45m15.969923198s) [upload=116117]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.292363469s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [............................X..X............................] duration: 8.966229708s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:45:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 8.966267493s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.......X....] duration: 1.337913694s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:45:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.338006731s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 900.541499ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:45:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 900.582047ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 1.38875311s trace: https://jaeger.dev.storj.io/trace/277ba948f183dc4f FAIL (2026-03-10 18:45:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:44:18.395709628 (103h44m15.992480658s) [upload=116117]: Starting upload Mar 10 18:44:18.395810068 (103h44m15.992581086s) [upload=116117]: Done starting upload Mar 10 18:44:18.395929782 (103h44m15.992700811s) [upload=116117]: Uploading segments... Mar 10 18:44:18.396421871 (103h44m15.993192902s) [upload=116117]: Got next segment. Inline: false Mar 10 18:44:18.396541809 (103h44m15.993312834s) [upload=116117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:44:18.396549171 (103h44m15.993320203s) [upload=116117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:44:18.428758532 (103h44m16.025529561s) [seg_pos={0 0}, upload=116117]: Begin upload segment... Mar 10 18:44:18.428822285 (103h44m16.0255933s) [seg_pos={0 0}, upload=116117]: Done begin upload segment. Mar 10 18:44:18.428861616 (103h44m16.02563264s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.429059881 (103h44m16.025830906s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430270915 (103h44m16.027041946s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430727660 (103h44m16.027498692s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431207878 (103h44m16.027978905s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431250249 (103h44m16.028021271s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431644829 (103h44m16.028415855s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431798421 (103h44m16.028569448s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432077244 (103h44m16.028848275s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432428854 (103h44m16.029199881s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432499385 (103h44m16.029270419s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432975554 (103h44m16.029746589s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.433599251 (103h44m16.030370283s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434278354 (103h44m16.031049379s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434667011 (103h44m16.031438044s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435223698 (103h44m16.031994733s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435757687 (103h44m16.032528714s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.436241820 (103h44m16.033012853s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.512361085 (103h44m16.109132112s) [upload=116117]: Next returned nil segment Mar 10 18:44:18.512381685 (103h44m16.109152703s) [upload=116117]: Waiting for error group managing segments... Mar 10 18:44:18.516528033 (103h44m16.113299062s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.518829594 (103h44m16.115600608s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.519605999 (103h44m16.116377021s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533489430 (103h44m16.130260444s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533564325 (103h44m16.130335338s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.774929408 (103h44m16.371700437s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.792542126 (103h44m16.389313153s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.811444852 (103h44m16.408215877s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.009790186 (103h44m16.606561214s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.017479102 (103h44m16.614250127s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.032351604 (103h44m16.629122617s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.036028441 (103h44m16.632799471s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.038165250 (103h44m16.634936273s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:45:18.350922849 (103h45m15.947693883s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: write tcp 159.223.143.39:49376->77.90.31.13:4638: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351159799 (103h45m15.947930832s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 159.223.143.39:42258->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351372791 (103h45m15.948143825s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:46988->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351533653 (103h45m15.948304686s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 159.223.143.39:49664->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.352230657 (103h45m15.949001693s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:45:18.352270159 (103h45m15.949041186s) [seg_pos={0 0}, upload=116117]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:45:18.373031667 (103h45m15.9698027s) [upload=116117]: Done uploading segments... Mar 10 18:45:18.373152165 (103h45m15.969923198s) [upload=116117]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.388778366s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 1m0.025241136s trace: https://jaeger.dev.storj.io/trace/277ba948f183dc4f FAIL (2026-03-10 18:45:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:44:18.395709628 (103h44m15.992480658s) [upload=116117]: Starting upload Mar 10 18:44:18.395810068 (103h44m15.992581086s) [upload=116117]: Done starting upload Mar 10 18:44:18.395929782 (103h44m15.992700811s) [upload=116117]: Uploading segments... Mar 10 18:44:18.396421871 (103h44m15.993192902s) [upload=116117]: Got next segment. Inline: false Mar 10 18:44:18.396541809 (103h44m15.993312834s) [upload=116117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:44:18.396549171 (103h44m15.993320203s) [upload=116117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:44:18.428758532 (103h44m16.025529561s) [seg_pos={0 0}, upload=116117]: Begin upload segment... Mar 10 18:44:18.428822285 (103h44m16.0255933s) [seg_pos={0 0}, upload=116117]: Done begin upload segment. Mar 10 18:44:18.428861616 (103h44m16.02563264s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.429059881 (103h44m16.025830906s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430270915 (103h44m16.027041946s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.430727660 (103h44m16.027498692s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431207878 (103h44m16.027978905s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431250249 (103h44m16.028021271s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431644829 (103h44m16.028415855s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.431798421 (103h44m16.028569448s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432077244 (103h44m16.028848275s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432428854 (103h44m16.029199881s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432499385 (103h44m16.029270419s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.432975554 (103h44m16.029746589s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.433599251 (103h44m16.030370283s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434278354 (103h44m16.031049379s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.434667011 (103h44m16.031438044s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435223698 (103h44m16.031994733s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.435757687 (103h44m16.032528714s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.436241820 (103h44m16.033012853s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Uploading piece... Mar 10 18:44:18.512361085 (103h44m16.109132112s) [upload=116117]: Next returned nil segment Mar 10 18:44:18.512381685 (103h44m16.109152703s) [upload=116117]: Waiting for error group managing segments... Mar 10 18:44:18.516528033 (103h44m16.113299062s) [noise=true , address=79.127.219.46:4896 , piece_id=4O6WKTLPEMPINFFXKN2CSQ7Z6XS5DBLFWKQUO7Z6MY2BVVOBULPA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.518829594 (103h44m16.115600608s) [noise=true , address=145.79.227.47:4285 , piece_id=DD7BAQTMTDM7PVAYWVW4AAWKEMQYNPQYMOEYLGXEXSW6TQIYFQTQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.519605999 (103h44m16.116377021s) [noise=true , address=145.79.227.11:4758 , piece_id=XTAARLVCHCWU3CR4PCVLNI3YTKIWKDU7EFSNULZULWGH3X3GZEJA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533489430 (103h44m16.130260444s) [noise=true , address=45.91.116.11:4133 , piece_id=X7HHRJHVCCARXSB3S7IWVEVHQCSMXD6PUXHFNK4XFZY4QCIOZLEA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.533564325 (103h44m16.130335338s) [noise=true , address=45.91.116.8:4056 , piece_id=5DAIGDMLWMSEJE5YU3JVY7PC4KDNMTBSEODO2TQGHYZO6NEKS25Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.774929408 (103h44m16.371700437s) [noise=true , address=167.17.72.68:4668 , piece_id=4SHLW7OPM6HOAP7FTSEIGB2Z5QWRSTT4BY7GQZQ74ME4NQVOA2IQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.792542126 (103h44m16.389313153s) [noise=true , address=167.17.72.70:4780 , piece_id=5FEPNFICLL7AFFJFBGN3DDFO44WIHDYS4ZPYGA4ZWZNUWPNREWGA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:18.811444852 (103h44m16.408215877s) [noise=true , address=167.17.72.69:4712 , piece_id=77IHJBDGRFBAKGFGQBRW54XNL6ZMRLJP55AAIXFDN7WM3OMAW7AA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.009790186 (103h44m16.606561214s) [noise=true , address=79.127.205.233:4171 , piece_id=XVWZ3SPDZFNCCCEM7JRUPDXLEH7ENNRBJE6NGWQ3JWFEOTMXLJKA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.017479102 (103h44m16.614250127s) [noise=true , address=193.149.164.69:4078 , piece_id=DJ6VJXG5CTSL4FN2Y7Z34IR7QHZM6F7OYQKZVTCXQHM7EKYJMLSQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.032351604 (103h44m16.629122617s) [noise=true , address=193.149.164.70:4111 , piece_id=PPKYJGDP4IPIGPN3LPB5JC26G2KXS3FGUOOWYUCXO66VX3PNSVBQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.036028441 (103h44m16.632799471s) [noise=true , address=193.149.164.67:4005 , piece_id=VJW7P6OTRS7VLW7OHSVQP3KDGOK7X2KQALCRYNSCVAASXMDWCIZA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:44:19.038165250 (103h44m16.634936273s) [noise=true , address=79.127.205.237:4214 , piece_id=JPKEUT4QZJFKYE55A3CNNF3WP77MLIJ5CO2HMQDMQSQTZTI4FA7Q, seg_pos={0 0}, upload=116117]: Done uploading piece. err: <nil> Mar 10 18:45:18.350922849 (103h45m15.947693883s) [noise=true , address=77.90.31.13:4638 , piece_id=AGWOYO3V3PDK7EWFXAL7Y3WKA6OA4EQ62KC2FY6MRP2JQLVQESUQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: write tcp 159.223.143.39:49376->77.90.31.13:4638: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351159799 (103h45m15.947930832s) [noise=true , address=77.90.31.14:4695 , piece_id=DOLOZHONA5PKQX777AZF2Y554NHUVKPXGRFAV3Q4YSJAAG5PI7XQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: write tcp 159.223.143.39:42258->77.90.31.14:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351372791 (103h45m15.948143825s) [noise=true , address=77.90.31.11:4574 , piece_id=ETSPHANVETQZBGEVJ5MAUIYH735ZQUIKMDFEZCBR46GY6LNNC3LA, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:46988->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.351533653 (103h45m15.948304686s) [noise=true , address=77.90.31.17:4798 , piece_id=RJBBLPFKIN2EXSTVP6EEA6RWFKF5VDHOCNVMAMKL6BX3JNR6DS4A, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 159.223.143.39:49664->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:45:18.352230657 (103h45m15.949001693s) [noise=true , address=77.90.31.15:4718 , piece_id=2CO36VS5NTAJEJPU24YQLQA6MNTNEUQFOWAUS72S2K7XCEPFNLZQ, seg_pos={0 0}, upload=116117]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:45:18.352270159 (103h45m15.949041186s) [seg_pos={0 0}, upload=116117]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:45:18.373031667 (103h45m15.9698027s) [upload=116117]: Done uploading segments... Mar 10 18:45:18.373152165 (103h45m15.969923198s) [upload=116117]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.02526595s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..................X..] duration: 1.122469889s trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:45:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.122530226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..X...........................] duration: 4.731113531s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:45:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 4.731188459s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.......X...] duration: 1.24868702s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.24872277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 524.442365ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:44:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 524.468436ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..X..........................] duration: 3.3840413s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:44:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 3.384116897s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..................X.] duration: 810.409181ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:44:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 810.516613ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.429144738s trace: https://jaeger.dev.storj.io/trace/2c09dc74a039dd47 FAIL (2026-03-10 18:43:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:41:41.913748936 (103h41m39.510519969s) [upload=116070]: Starting upload Mar 10 18:41:41.913823998 (103h41m39.510595022s) [upload=116070]: Done starting upload Mar 10 18:41:41.914018879 (103h41m39.510789904s) [upload=116070]: Uploading segments... Mar 10 18:41:41.914292096 (103h41m39.511063126s) [upload=116070]: Got next segment. Inline: false Mar 10 18:41:41.914358591 (103h41m39.511129613s) [upload=116070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:41:41.914364810 (103h41m39.511135837s) [upload=116070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:41:41.945876020 (103h41m39.542647047s) [seg_pos={0 0}, upload=116070]: Begin upload segment... Mar 10 18:41:41.945989941 (103h41m39.542760956s) [seg_pos={0 0}, upload=116070]: Done begin upload segment. Mar 10 18:41:41.946038463 (103h41m39.542809492s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947267812 (103h41m39.544038844s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947404666 (103h41m39.544175694s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947688844 (103h41m39.544459888s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947920415 (103h41m39.544691441s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948073429 (103h41m39.544844458s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948351647 (103h41m39.54512267s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948412974 (103h41m39.545184s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948780959 (103h41m39.545551982s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948787318 (103h41m39.545558344s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949242022 (103h41m39.546013053s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949426769 (103h41m39.546197797s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949622507 (103h41m39.546393531s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949915157 (103h41m39.546686187s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950446990 (103h41m39.547217999s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950938479 (103h41m39.547709503s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951430628 (103h41m39.54820166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951902825 (103h41m39.548673851s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:42.018268845 (103h41m39.615039878s) [upload=116070]: Next returned nil segment Mar 10 18:41:42.018288541 (103h41m39.615059577s) [upload=116070]: Waiting for error group managing segments... Mar 10 18:41:42.031824422 (103h41m39.628595443s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.032133163 (103h41m39.628904176s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.037205272 (103h41m39.633976294s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.041783230 (103h41m39.638554254s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.044401132 (103h41m39.641172147s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.050819140 (103h41m39.647590166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.060650496 (103h41m39.65742152s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.290598131 (103h41m39.887369166s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.324195882 (103h41m39.920966895s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.495666794 (103h41m40.092437807s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.527834063 (103h41m40.124605089s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.537057440 (103h41m40.133828464s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.628608162 (103h41m40.22537918s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:42:41.883186070 (103h42m39.479957103s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6, address:77.90.31.18:4818): protocol: rpc: dial tcp 77.90.31.18:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883366529 (103h42m39.480137564s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, address:77.90.31.19:4862): protocol: write tcp 159.223.143.39:55634->77.90.31.19:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883420336 (103h42m39.480191362s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: rpc: dial tcp 77.90.31.16:4741: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883765681 (103h42m39.480536704s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: write tcp 159.223.143.39:48342->77.90.31.13:4635: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883877856 (103h42m39.48064889s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, address:77.90.31.15:4706): protocol: write tcp 159.223.143.39:35730->77.90.31.15:4706: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883913013 (103h42m39.480684023s) [seg_pos={0 0}, upload=116070]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:42:41.908852084 (103h42m39.505623111s) [upload=116070]: Done uploading segments... Mar 10 18:42:41.908901765 (103h42m39.505672794s) [upload=116070]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.429171266s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..................X] duration: 1m0.002448161s trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:43:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002482477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.......X..] duration: 1.248444775s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.24847008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..X.........................] duration: 6.315023717s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:43:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.315046966s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 768.206657ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:43:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 768.253352ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.508323939s trace: https://jaeger.dev.storj.io/trace/2c09dc74a039dd47 FAIL (2026-03-10 18:43:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:41:41.913748936 (103h41m39.510519969s) [upload=116070]: Starting upload Mar 10 18:41:41.913823998 (103h41m39.510595022s) [upload=116070]: Done starting upload Mar 10 18:41:41.914018879 (103h41m39.510789904s) [upload=116070]: Uploading segments... Mar 10 18:41:41.914292096 (103h41m39.511063126s) [upload=116070]: Got next segment. Inline: false Mar 10 18:41:41.914358591 (103h41m39.511129613s) [upload=116070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:41:41.914364810 (103h41m39.511135837s) [upload=116070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:41:41.945876020 (103h41m39.542647047s) [seg_pos={0 0}, upload=116070]: Begin upload segment... Mar 10 18:41:41.945989941 (103h41m39.542760956s) [seg_pos={0 0}, upload=116070]: Done begin upload segment. Mar 10 18:41:41.946038463 (103h41m39.542809492s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947267812 (103h41m39.544038844s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947404666 (103h41m39.544175694s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947688844 (103h41m39.544459888s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947920415 (103h41m39.544691441s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948073429 (103h41m39.544844458s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948351647 (103h41m39.54512267s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948412974 (103h41m39.545184s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948780959 (103h41m39.545551982s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948787318 (103h41m39.545558344s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949242022 (103h41m39.546013053s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949426769 (103h41m39.546197797s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949622507 (103h41m39.546393531s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949915157 (103h41m39.546686187s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950446990 (103h41m39.547217999s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950938479 (103h41m39.547709503s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951430628 (103h41m39.54820166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951902825 (103h41m39.548673851s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:42.018268845 (103h41m39.615039878s) [upload=116070]: Next returned nil segment Mar 10 18:41:42.018288541 (103h41m39.615059577s) [upload=116070]: Waiting for error group managing segments... Mar 10 18:41:42.031824422 (103h41m39.628595443s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.032133163 (103h41m39.628904176s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.037205272 (103h41m39.633976294s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.041783230 (103h41m39.638554254s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.044401132 (103h41m39.641172147s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.050819140 (103h41m39.647590166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.060650496 (103h41m39.65742152s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.290598131 (103h41m39.887369166s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.324195882 (103h41m39.920966895s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.495666794 (103h41m40.092437807s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.527834063 (103h41m40.124605089s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.537057440 (103h41m40.133828464s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.628608162 (103h41m40.22537918s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:42:41.883186070 (103h42m39.479957103s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6, address:77.90.31.18:4818): protocol: rpc: dial tcp 77.90.31.18:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883366529 (103h42m39.480137564s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, address:77.90.31.19:4862): protocol: write tcp 159.223.143.39:55634->77.90.31.19:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883420336 (103h42m39.480191362s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: rpc: dial tcp 77.90.31.16:4741: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883765681 (103h42m39.480536704s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: write tcp 159.223.143.39:48342->77.90.31.13:4635: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883877856 (103h42m39.48064889s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, address:77.90.31.15:4706): protocol: write tcp 159.223.143.39:35730->77.90.31.15:4706: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883913013 (103h42m39.480684023s) [seg_pos={0 0}, upload=116070]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:42:41.908852084 (103h42m39.505623111s) [upload=116070]: Done uploading segments... Mar 10 18:42:41.908901765 (103h42m39.505672794s) [upload=116070]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.508363674s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration: 1.103175445s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:43:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.103193489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 1m0.028478074s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:43:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.028490535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 573.921791ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:42:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 573.948249ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 860.961632ms trace: https://jaeger.dev.storj.io/trace/2c09dc74a039dd47 FAIL (2026-03-10 18:42:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:41:41.913748936 (103h41m39.510519969s) [upload=116070]: Starting upload Mar 10 18:41:41.913823998 (103h41m39.510595022s) [upload=116070]: Done starting upload Mar 10 18:41:41.914018879 (103h41m39.510789904s) [upload=116070]: Uploading segments... Mar 10 18:41:41.914292096 (103h41m39.511063126s) [upload=116070]: Got next segment. Inline: false Mar 10 18:41:41.914358591 (103h41m39.511129613s) [upload=116070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:41:41.914364810 (103h41m39.511135837s) [upload=116070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:41:41.945876020 (103h41m39.542647047s) [seg_pos={0 0}, upload=116070]: Begin upload segment... Mar 10 18:41:41.945989941 (103h41m39.542760956s) [seg_pos={0 0}, upload=116070]: Done begin upload segment. Mar 10 18:41:41.946038463 (103h41m39.542809492s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947267812 (103h41m39.544038844s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947404666 (103h41m39.544175694s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947688844 (103h41m39.544459888s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947920415 (103h41m39.544691441s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948073429 (103h41m39.544844458s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948351647 (103h41m39.54512267s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948412974 (103h41m39.545184s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948780959 (103h41m39.545551982s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948787318 (103h41m39.545558344s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949242022 (103h41m39.546013053s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949426769 (103h41m39.546197797s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949622507 (103h41m39.546393531s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949915157 (103h41m39.546686187s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950446990 (103h41m39.547217999s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950938479 (103h41m39.547709503s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951430628 (103h41m39.54820166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951902825 (103h41m39.548673851s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:42.018268845 (103h41m39.615039878s) [upload=116070]: Next returned nil segment Mar 10 18:41:42.018288541 (103h41m39.615059577s) [upload=116070]: Waiting for error group managing segments... Mar 10 18:41:42.031824422 (103h41m39.628595443s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.032133163 (103h41m39.628904176s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.037205272 (103h41m39.633976294s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.041783230 (103h41m39.638554254s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.044401132 (103h41m39.641172147s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.050819140 (103h41m39.647590166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.060650496 (103h41m39.65742152s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.290598131 (103h41m39.887369166s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.324195882 (103h41m39.920966895s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.495666794 (103h41m40.092437807s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.527834063 (103h41m40.124605089s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.537057440 (103h41m40.133828464s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.628608162 (103h41m40.22537918s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:42:41.883186070 (103h42m39.479957103s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6, address:77.90.31.18:4818): protocol: rpc: dial tcp 77.90.31.18:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883366529 (103h42m39.480137564s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, address:77.90.31.19:4862): protocol: write tcp 159.223.143.39:55634->77.90.31.19:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883420336 (103h42m39.480191362s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: rpc: dial tcp 77.90.31.16:4741: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883765681 (103h42m39.480536704s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: write tcp 159.223.143.39:48342->77.90.31.13:4635: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883877856 (103h42m39.48064889s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, address:77.90.31.15:4706): protocol: write tcp 159.223.143.39:35730->77.90.31.15:4706: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883913013 (103h42m39.480684023s) [seg_pos={0 0}, upload=116070]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:42:41.908852084 (103h42m39.505623111s) [upload=116070]: Done uploading segments... Mar 10 18:42:41.908901765 (103h42m39.505672794s) [upload=116070]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 860.982824ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.028388088s trace: https://jaeger.dev.storj.io/trace/2c09dc74a039dd47 FAIL (2026-03-10 18:42:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:41:41.913748936 (103h41m39.510519969s) [upload=116070]: Starting upload Mar 10 18:41:41.913823998 (103h41m39.510595022s) [upload=116070]: Done starting upload Mar 10 18:41:41.914018879 (103h41m39.510789904s) [upload=116070]: Uploading segments... Mar 10 18:41:41.914292096 (103h41m39.511063126s) [upload=116070]: Got next segment. Inline: false Mar 10 18:41:41.914358591 (103h41m39.511129613s) [upload=116070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:41:41.914364810 (103h41m39.511135837s) [upload=116070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:41:41.945876020 (103h41m39.542647047s) [seg_pos={0 0}, upload=116070]: Begin upload segment... Mar 10 18:41:41.945989941 (103h41m39.542760956s) [seg_pos={0 0}, upload=116070]: Done begin upload segment. Mar 10 18:41:41.946038463 (103h41m39.542809492s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947267812 (103h41m39.544038844s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947404666 (103h41m39.544175694s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947688844 (103h41m39.544459888s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.947920415 (103h41m39.544691441s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948073429 (103h41m39.544844458s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948351647 (103h41m39.54512267s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948412974 (103h41m39.545184s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948780959 (103h41m39.545551982s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.948787318 (103h41m39.545558344s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949242022 (103h41m39.546013053s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949426769 (103h41m39.546197797s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949622507 (103h41m39.546393531s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.949915157 (103h41m39.546686187s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950446990 (103h41m39.547217999s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.950938479 (103h41m39.547709503s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951430628 (103h41m39.54820166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:41.951902825 (103h41m39.548673851s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Uploading piece... Mar 10 18:41:42.018268845 (103h41m39.615039878s) [upload=116070]: Next returned nil segment Mar 10 18:41:42.018288541 (103h41m39.615059577s) [upload=116070]: Waiting for error group managing segments... Mar 10 18:41:42.031824422 (103h41m39.628595443s) [noise=true , address=45.91.116.5:4319 , piece_id=GGOSD3ZFV7O3SHAAV4HOJ5NMSMUOK7DWNWHHCTGYFCWSMRMZ3BHA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.032133163 (103h41m39.628904176s) [noise=true , address=45.91.116.12:4248 , piece_id=TTPKHR2ZWHR4TSBY3UNV2FFI7QGL5ATUWIBBCM4MIICGGHC3GJXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.037205272 (103h41m39.633976294s) [noise=true , address=145.79.227.14:4866 , piece_id=MJWGET2KFZUMZXKF6V6GIUWKMX4LRXZV6POBFBXE4R6KSKHD56BA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.041783230 (103h41m39.638554254s) [noise=true , address=45.91.116.11:4122 , piece_id=7Z2TUTXKRMNARGF45FTHGXFNOHF4LSTOY4V4T7L4XQ3O6Z6AG24Q, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.044401132 (103h41m39.641172147s) [noise=true , address=79.127.219.45:4860 , piece_id=BMKKIGRZNNKP56SRVFMXVQXTHPAXDAQSYXSMNGIUFSCPW45E3IWQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.050819140 (103h41m39.647590166s) [noise=true , address=79.127.219.38:4829 , piece_id=JT2C4S3KJVP7YTRE2O6ZAPDGNKZGUVUABCTB2SJRMV45TNY57BXA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.060650496 (103h41m39.65742152s) [noise=true , address=145.79.227.26:4683 , piece_id=RPGKWA3KZR5SXG3VYLX24DWYPVUX7FOZIAC2BGVZ52NRTDOVWHMQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.290598131 (103h41m39.887369166s) [noise=true , address=167.17.72.68:4664 , piece_id=4YBLETUDPCFLKBZNIDSHNKGH42EWZ6WTMEBT7JBZ3TP4LYK6FHNA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.324195882 (103h41m39.920966895s) [noise=true , address=167.17.72.69:4729 , piece_id=FG3T53WSS24O5PZYPUZOP5BXQ4ATABBWJZUHJYOWZCP7E6S4JASQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.495666794 (103h41m40.092437807s) [noise=true , address=79.127.205.233:4150 , piece_id=YG3IT3V5LDDVFFFKLVHADB5FJQQGAYXX3WIP7MEW6EXHIKKIA6RA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.527834063 (103h41m40.124605089s) [noise=true , address=193.149.164.69:4069 , piece_id=5QKTK2BUUVJONL5VYS5ZVPFFGM6FYBTL3AEWMZB47DWN72YF3XQA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.537057440 (103h41m40.133828464s) [noise=true , address=193.149.164.70:4106 , piece_id=F6WHE2GMYULRL4FYYUNJEZ5NGXCIJJWZP7REUFJFIHLHUHMB6SEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:41:42.628608162 (103h41m40.22537918s) [noise=true , address=64.49.13.169:4714 , piece_id=F6EH5KSZP2RDY35AES2VPZ2GE6C3M37WWMM3Y3CAXBWCPX2WLI7A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: <nil> Mar 10 18:42:41.883186070 (103h42m39.479957103s) [noise=true , address=77.90.31.18:4818 , piece_id=GBHU3LHOCCD3PW7JZJZB27VJ7QA727H3P6YIWLDEV4BJGEQCD35A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12LuFBTJg1aNmZoEdiQTtvfnfh5qnWVZFtqFX93L4fNfiYTYqJ6, address:77.90.31.18:4818): protocol: rpc: dial tcp 77.90.31.18:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883366529 (103h42m39.480137564s) [noise=true , address=77.90.31.19:4862 , piece_id=HZTYX4CX6W3UAYQBUAQBO3S6KA75V6UM43S27SGY6NJBCCT5NM3A, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, address:77.90.31.19:4862): protocol: write tcp 159.223.143.39:55634->77.90.31.19:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883420336 (103h42m39.480191362s) [noise=true , address=77.90.31.16:4741 , piece_id=JXU6TCX7UMYQHY3EJ3H4JR4XULZC5CGMZG3JTJV3YJIMABX2YNSA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: rpc: dial tcp 77.90.31.16:4741: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883765681 (103h42m39.480536704s) [noise=true , address=77.90.31.13:4635 , piece_id=CZ3SI45HOXDDEWUSUA5RQ5CTFOVW3KQTEMA6OW64D5I3MZMDXPEA, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: write tcp 159.223.143.39:48342->77.90.31.13:4635: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883877856 (103h42m39.48064889s) [noise=true , address=77.90.31.15:4706 , piece_id=W7NRCLFCQE7XLP34VSUMWEI7KUCXCE2BCTYQJPLC6E4JCCWW4OXQ, seg_pos={0 0}, upload=116070]: Done uploading piece. err: ecclient: upload failed (node:12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, address:77.90.31.15:4706): protocol: write tcp 159.223.143.39:35730->77.90.31.15:4706: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:42:41.883913013 (103h42m39.480684023s) [seg_pos={0 0}, upload=116070]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:42:41.908852084 (103h42m39.505623111s) [upload=116070]: Done uploading segments... Mar 10 18:42:41.908901765 (103h42m39.505672794s) [upload=116070]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.028468278s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [................................X..X........................] duration: 3.624967354s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:42:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 3.624999413s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 618.30668ms trace: https://jaeger.dev.storj.io/trace/62ea03c3a72aaf43 FAIL (2026-03-10 18:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 618.519589ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 537.98726ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:42:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.038043ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..X.......................] duration: 2.617772656s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:41:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.617791789s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..X......................] duration: 5.282122295s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:41:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.282151056s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 507.06736ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:41:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.094032ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.229643552s trace: https://jaeger.dev.storj.io/trace/14d851485ec55b94 FAIL (2026-03-10 18:41:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== egments... Mar 10 18:34:13.234105804 (103h34m10.830876822s) [upload=115928]: Got next segment. Inline: false Mar 10 18:34:13.234171800 (103h34m10.830942823s) [upload=115928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:34:13.234177690 (103h34m10.830948712s) [upload=115928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:34:13.264565602 (103h34m10.861336626s) [seg_pos={0 0}, upload=115928]: Begin upload segment... Mar 10 18:34:13.264650554 (103h34m10.861421568s) [seg_pos={0 0}, upload=115928]: Done begin upload segment. Mar 10 18:34:13.264710110 (103h34m10.861481134s) [noise=true , address=45.91.116.9:4280 , piece_id=O6XIUDM5QEPO2S3TSGBRWFJOA6YNOPISRIKOKJMYIYVCY3W3O6FA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.265998754 (103h34m10.862769779s) [noise=true , address=77.90.31.14:4669 , piece_id=P7P7VFJ2CX2EXILR5AVF6ZQJSRWJ7HFBWB6APJUKQOETOPO7N3CQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.266563685 (103h34m10.863334713s) [noise=true , address=77.90.31.19:4863 , piece_id=DWBZ64GZUGWORUXIZGZVHTWA3VAB4GILLMMXVU5KB2VBGRX7FN2Q, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.266980797 (103h34m10.863751825s) [noise=true , address=193.149.164.68:4044 , piece_id=HBSJYACUD45I5CQBQ3PJZOWN4MBTXJCTGHMKHOMTUG5YCDISGPOA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267514292 (103h34m10.864285328s) [noise=true , address=79.127.205.235:4262 , piece_id=X6XAWZ64BAJP2EYT4PXLQHP6NTPCMRXSGHPTDF55V4Z3XXUZVO7A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267576292 (103h34m10.864347316s) [noise=true , address=193.149.164.69:4071 , piece_id=WVYSHF5Z35Z4DFCEVWVBKRRDLELXAV5URWMSFHNKCYCBQBKXBXIA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267944808 (103h34m10.864715836s) [noise=true , address=193.149.164.67:4032 , piece_id=OYVBK5VQX2ICUTG6LTGB5DOEUKWA2Q42ANILPBUA3ELWA5XZ7OSQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268291059 (103h34m10.865062093s) [noise=true , address=79.127.205.234:4190 , piece_id=2DCWY6MVVYBU7R5R2G6LABDDL6YCBCYEQI3B5GL2P67SFCFU7ECQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268635689 (103h34m10.865406733s) [noise=true , address=79.127.205.237:4241 , piece_id=EMX5GFPZX3VLRD4AG3SDATNVA3YSJ6EAB4WXY3LTFE6AWEO7XZ4A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268857349 (103h34m10.865628383s) [noise=true , address=77.90.31.11:4587 , piece_id=XDEV3CN5NW3VMYC7B3HYF2YLSQ6GV334LG2YYD2X4PIJQE3ZNEYA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.269379163 (103h34m10.866150189s) [noise=true , address=145.79.227.31:4822 , piece_id=7A2BFMPYL6EWUQC5RFZY6IIVXIVQLML6TSZVUP7VU4NIQLWYB63Q, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.269442067 (103h34m10.866213117s) [noise=true , address=77.90.31.13:4647 , piece_id=5DBAZJVJSJZF2AHFPAQBVTDKTF4LRXTJRNNLMSNDHD6G5TY366GQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.270023291 (103h34m10.866794323s) [noise=true , address=193.149.164.71:4298 , piece_id=VMRPXXIJIKZAWY6SCIA3FMAFIQ4XYI4LWXLVFRNMSKY5B6MV5DMQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.270558766 (103h34m10.867329801s) [noise=true , address=79.127.219.41:4954 , piece_id=UMD2OKNI36MLUAC5TOF6PVK573GN3JJSM3HXCYOLG25EGRPDVHJA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271066172 (103h34m10.867837205s) [noise=true , address=77.90.31.18:4837 , piece_id=N5RMIIIGCAQES6G6YXHCUX2YRVKIE764F57XQFGWOCKYOBJV33ZA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271143045 (103h34m10.867914074s) [noise=true , address=167.17.72.68:4645 , piece_id=76HO76ZYEW7YE2XGOZY3EJJHTZFW24B7IFGF3IMUV6IMYKTHLG2A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271561296 (103h34m10.86833233s) [noise=true , address=167.17.72.70:4755 , piece_id=NXGZWBH2UNVNSE7B2BHVBEC434N7MWHDYTDECVTZNB3O6GYPZUNA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.272067991 (103h34m10.868839022s) [noise=true , address=79.127.219.45:4878 , piece_id=GTM5BKXU6TQKMBCF5FOW34OKWVQLUJUJ4WINDYBZFR3ORI2OSHSA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.341870072 (103h34m10.938641097s) [upload=115928]: Next returned nil segment Mar 10 18:34:13.341887354 (103h34m10.938658381s) [upload=115928]: Waiting for error group managing segments... Mar 10 18:34:13.357317608 (103h34m10.954088641s) [noise=true , address=79.127.219.41:4954 , piece_id=UMD2OKNI36MLUAC5TOF6PVK573GN3JJSM3HXCYOLG25EGRPDVHJA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.369079638 (103h34m10.965850664s) [noise=true , address=45.91.116.9:4280 , piece_id=O6XIUDM5QEPO2S3TSGBRWFJOA6YNOPISRIKOKJMYIYVCY3W3O6FA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.383782380 (103h34m10.980553407s) [noise=true , address=145.79.227.31:4822 , piece_id=7A2BFMPYL6EWUQC5RFZY6IIVXIVQLML6TSZVUP7VU4NIQLWYB63Q, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.383844707 (103h34m10.980615732s) [noise=true , address=79.127.219.45:4878 , piece_id=GTM5BKXU6TQKMBCF5FOW34OKWVQLUJUJ4WINDYBZFR3ORI2OSHSA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.596292826 (103h34m11.19306384s) [noise=true , address=167.17.72.68:4645 , piece_id=76HO76ZYEW7YE2XGOZY3EJJHTZFW24B7IFGF3IMUV6IMYKTHLG2A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.657877077 (103h34m11.254648103s) [noise=true , address=167.17.72.70:4755 , piece_id=NXGZWBH2UNVNSE7B2BHVBEC434N7MWHDYTDECVTZNB3O6GYPZUNA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.815668472 (103h34m11.412439501s) [noise=true , address=79.127.205.234:4190 , piece_id=2DCWY6MVVYBU7R5R2G6LABDDL6YCBCYEQI3B5GL2P67SFCFU7ECQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.816829420 (103h34m11.413600446s) [noise=true , address=79.127.205.237:4241 , piece_id=EMX5GFPZX3VLRD4AG3SDATNVA3YSJ6EAB4WXY3LTFE6AWEO7XZ4A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.818063674 (103h34m11.414834697s) [noise=true , address=79.127.205.235:4262 , piece_id=X6XAWZ64BAJP2EYT4PXLQHP6NTPCMRXSGHPTDF55V4Z3XXUZVO7A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.844922561 (103h34m11.441693585s) [noise=true , address=193.149.164.67:4032 , piece_id=OYVBK5VQX2ICUTG6LTGB5DOEUKWA2Q42ANILPBUA3ELWA5XZ7OSQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.862417635 (103h34m11.459188668s) [noise=true , address=193.149.164.71:4298 , piece_id=VMRPXXIJIKZAWY6SCIA3FMAFIQ4XYI4LWXLVFRNMSKY5B6MV5DMQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.864932308 (103h34m11.461703323s) [noise=true , address=193.149.164.69:4071 , piece_id=WVYSHF5Z35Z4DFCEVWVBKRRDLELXAV5URWMSFHNKCYCBQBKXBXIA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.874686150 (103h34m11.471457173s) [noise=true , address=193.149.164.68:4044 , piece_id=HBSJYACUD45I5CQBQ3PJZOWN4MBTXJCTGHMKHOMTUG5YCDISGPOA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:35:13.203204147 (103h35m10.799975179s) [noise=true , address=77.90.31.19:4863 , piece_id=DWBZ64GZUGWORUXIZGZVHTWA3VAB4GILLMMXVU5KB2VBGRX7FN2Q, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: rpc: dial tcp 77.90.31.19:4863: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203494517 (103h35m10.800265546s) [noise=true , address=77.90.31.18:4837 , piece_id=N5RMIIIGCAQES6G6YXHCUX2YRVKIE764F57XQFGWOCKYOBJV33ZA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: write tcp 159.223.143.39:56048->77.90.31.18:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203740816 (103h35m10.800511848s) [noise=true , address=77.90.31.13:4647 , piece_id=5DBAZJVJSJZF2AHFPAQBVTDKTF4LRXTJRNNLMSNDHD6G5TY366GQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: rpc: dial tcp 77.90.31.13:4647: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203754450 (103h35m10.800525479s) [noise=true , address=77.90.31.11:4587 , piece_id=XDEV3CN5NW3VMYC7B3HYF2YLSQ6GV334LG2YYD2X4PIJQE3ZNEYA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: rpc: dial tcp 77.90.31.11:4587: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.204001069 (103h35m10.800772099s) [noise=true , address=77.90.31.14:4669 , piece_id=P7P7VFJ2CX2EXILR5AVF6ZQJSRWJ7HFBWB6APJUKQOETOPO7N3CQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.204030719 (103h35m10.800801732s) [seg_pos={0 0}, upload=115928]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:35:13.239310552 (103h35m10.836081578s) [upload=115928]: Done uploading segments... Mar 10 18:35:13.239351234 (103h35m10.836122261s) [upload=115928]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.229669603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 614.960942ms trace: https://jaeger.dev.storj.io/trace/62ea03c3a72aaf43 FAIL (2026-03-10 18:41:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 614.988914ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..X.....................] duration: 6.045073201s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:41:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.04510459s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 603.263599ms trace: https://jaeger.dev.storj.io/trace/62ea03c3a72aaf43 FAIL (2026-03-10 18:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 603.301466ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 605.374317ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:40:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.402655ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.304051307s trace: https://jaeger.dev.storj.io/trace/14d851485ec55b94 FAIL (2026-03-10 18:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== egments... Mar 10 18:34:13.234105804 (103h34m10.830876822s) [upload=115928]: Got next segment. Inline: false Mar 10 18:34:13.234171800 (103h34m10.830942823s) [upload=115928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:34:13.234177690 (103h34m10.830948712s) [upload=115928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:34:13.264565602 (103h34m10.861336626s) [seg_pos={0 0}, upload=115928]: Begin upload segment... Mar 10 18:34:13.264650554 (103h34m10.861421568s) [seg_pos={0 0}, upload=115928]: Done begin upload segment. Mar 10 18:34:13.264710110 (103h34m10.861481134s) [noise=true , address=45.91.116.9:4280 , piece_id=O6XIUDM5QEPO2S3TSGBRWFJOA6YNOPISRIKOKJMYIYVCY3W3O6FA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.265998754 (103h34m10.862769779s) [noise=true , address=77.90.31.14:4669 , piece_id=P7P7VFJ2CX2EXILR5AVF6ZQJSRWJ7HFBWB6APJUKQOETOPO7N3CQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.266563685 (103h34m10.863334713s) [noise=true , address=77.90.31.19:4863 , piece_id=DWBZ64GZUGWORUXIZGZVHTWA3VAB4GILLMMXVU5KB2VBGRX7FN2Q, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.266980797 (103h34m10.863751825s) [noise=true , address=193.149.164.68:4044 , piece_id=HBSJYACUD45I5CQBQ3PJZOWN4MBTXJCTGHMKHOMTUG5YCDISGPOA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267514292 (103h34m10.864285328s) [noise=true , address=79.127.205.235:4262 , piece_id=X6XAWZ64BAJP2EYT4PXLQHP6NTPCMRXSGHPTDF55V4Z3XXUZVO7A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267576292 (103h34m10.864347316s) [noise=true , address=193.149.164.69:4071 , piece_id=WVYSHF5Z35Z4DFCEVWVBKRRDLELXAV5URWMSFHNKCYCBQBKXBXIA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267944808 (103h34m10.864715836s) [noise=true , address=193.149.164.67:4032 , piece_id=OYVBK5VQX2ICUTG6LTGB5DOEUKWA2Q42ANILPBUA3ELWA5XZ7OSQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268291059 (103h34m10.865062093s) [noise=true , address=79.127.205.234:4190 , piece_id=2DCWY6MVVYBU7R5R2G6LABDDL6YCBCYEQI3B5GL2P67SFCFU7ECQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268635689 (103h34m10.865406733s) [noise=true , address=79.127.205.237:4241 , piece_id=EMX5GFPZX3VLRD4AG3SDATNVA3YSJ6EAB4WXY3LTFE6AWEO7XZ4A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268857349 (103h34m10.865628383s) [noise=true , address=77.90.31.11:4587 , piece_id=XDEV3CN5NW3VMYC7B3HYF2YLSQ6GV334LG2YYD2X4PIJQE3ZNEYA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.269379163 (103h34m10.866150189s) [noise=true , address=145.79.227.31:4822 , piece_id=7A2BFMPYL6EWUQC5RFZY6IIVXIVQLML6TSZVUP7VU4NIQLWYB63Q, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.269442067 (103h34m10.866213117s) [noise=true , address=77.90.31.13:4647 , piece_id=5DBAZJVJSJZF2AHFPAQBVTDKTF4LRXTJRNNLMSNDHD6G5TY366GQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.270023291 (103h34m10.866794323s) [noise=true , address=193.149.164.71:4298 , piece_id=VMRPXXIJIKZAWY6SCIA3FMAFIQ4XYI4LWXLVFRNMSKY5B6MV5DMQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.270558766 (103h34m10.867329801s) [noise=true , address=79.127.219.41:4954 , piece_id=UMD2OKNI36MLUAC5TOF6PVK573GN3JJSM3HXCYOLG25EGRPDVHJA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271066172 (103h34m10.867837205s) [noise=true , address=77.90.31.18:4837 , piece_id=N5RMIIIGCAQES6G6YXHCUX2YRVKIE764F57XQFGWOCKYOBJV33ZA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271143045 (103h34m10.867914074s) [noise=true , address=167.17.72.68:4645 , piece_id=76HO76ZYEW7YE2XGOZY3EJJHTZFW24B7IFGF3IMUV6IMYKTHLG2A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271561296 (103h34m10.86833233s) [noise=true , address=167.17.72.70:4755 , piece_id=NXGZWBH2UNVNSE7B2BHVBEC434N7MWHDYTDECVTZNB3O6GYPZUNA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.272067991 (103h34m10.868839022s) [noise=true , address=79.127.219.45:4878 , piece_id=GTM5BKXU6TQKMBCF5FOW34OKWVQLUJUJ4WINDYBZFR3ORI2OSHSA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.341870072 (103h34m10.938641097s) [upload=115928]: Next returned nil segment Mar 10 18:34:13.341887354 (103h34m10.938658381s) [upload=115928]: Waiting for error group managing segments... Mar 10 18:34:13.357317608 (103h34m10.954088641s) [noise=true , address=79.127.219.41:4954 , piece_id=UMD2OKNI36MLUAC5TOF6PVK573GN3JJSM3HXCYOLG25EGRPDVHJA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.369079638 (103h34m10.965850664s) [noise=true , address=45.91.116.9:4280 , piece_id=O6XIUDM5QEPO2S3TSGBRWFJOA6YNOPISRIKOKJMYIYVCY3W3O6FA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.383782380 (103h34m10.980553407s) [noise=true , address=145.79.227.31:4822 , piece_id=7A2BFMPYL6EWUQC5RFZY6IIVXIVQLML6TSZVUP7VU4NIQLWYB63Q, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.383844707 (103h34m10.980615732s) [noise=true , address=79.127.219.45:4878 , piece_id=GTM5BKXU6TQKMBCF5FOW34OKWVQLUJUJ4WINDYBZFR3ORI2OSHSA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.596292826 (103h34m11.19306384s) [noise=true , address=167.17.72.68:4645 , piece_id=76HO76ZYEW7YE2XGOZY3EJJHTZFW24B7IFGF3IMUV6IMYKTHLG2A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.657877077 (103h34m11.254648103s) [noise=true , address=167.17.72.70:4755 , piece_id=NXGZWBH2UNVNSE7B2BHVBEC434N7MWHDYTDECVTZNB3O6GYPZUNA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.815668472 (103h34m11.412439501s) [noise=true , address=79.127.205.234:4190 , piece_id=2DCWY6MVVYBU7R5R2G6LABDDL6YCBCYEQI3B5GL2P67SFCFU7ECQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.816829420 (103h34m11.413600446s) [noise=true , address=79.127.205.237:4241 , piece_id=EMX5GFPZX3VLRD4AG3SDATNVA3YSJ6EAB4WXY3LTFE6AWEO7XZ4A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.818063674 (103h34m11.414834697s) [noise=true , address=79.127.205.235:4262 , piece_id=X6XAWZ64BAJP2EYT4PXLQHP6NTPCMRXSGHPTDF55V4Z3XXUZVO7A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.844922561 (103h34m11.441693585s) [noise=true , address=193.149.164.67:4032 , piece_id=OYVBK5VQX2ICUTG6LTGB5DOEUKWA2Q42ANILPBUA3ELWA5XZ7OSQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.862417635 (103h34m11.459188668s) [noise=true , address=193.149.164.71:4298 , piece_id=VMRPXXIJIKZAWY6SCIA3FMAFIQ4XYI4LWXLVFRNMSKY5B6MV5DMQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.864932308 (103h34m11.461703323s) [noise=true , address=193.149.164.69:4071 , piece_id=WVYSHF5Z35Z4DFCEVWVBKRRDLELXAV5URWMSFHNKCYCBQBKXBXIA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.874686150 (103h34m11.471457173s) [noise=true , address=193.149.164.68:4044 , piece_id=HBSJYACUD45I5CQBQ3PJZOWN4MBTXJCTGHMKHOMTUG5YCDISGPOA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:35:13.203204147 (103h35m10.799975179s) [noise=true , address=77.90.31.19:4863 , piece_id=DWBZ64GZUGWORUXIZGZVHTWA3VAB4GILLMMXVU5KB2VBGRX7FN2Q, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: rpc: dial tcp 77.90.31.19:4863: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203494517 (103h35m10.800265546s) [noise=true , address=77.90.31.18:4837 , piece_id=N5RMIIIGCAQES6G6YXHCUX2YRVKIE764F57XQFGWOCKYOBJV33ZA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: write tcp 159.223.143.39:56048->77.90.31.18:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203740816 (103h35m10.800511848s) [noise=true , address=77.90.31.13:4647 , piece_id=5DBAZJVJSJZF2AHFPAQBVTDKTF4LRXTJRNNLMSNDHD6G5TY366GQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: rpc: dial tcp 77.90.31.13:4647: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203754450 (103h35m10.800525479s) [noise=true , address=77.90.31.11:4587 , piece_id=XDEV3CN5NW3VMYC7B3HYF2YLSQ6GV334LG2YYD2X4PIJQE3ZNEYA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: rpc: dial tcp 77.90.31.11:4587: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.204001069 (103h35m10.800772099s) [noise=true , address=77.90.31.14:4669 , piece_id=P7P7VFJ2CX2EXILR5AVF6ZQJSRWJ7HFBWB6APJUKQOETOPO7N3CQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.204030719 (103h35m10.800801732s) [seg_pos={0 0}, upload=115928]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:35:13.239310552 (103h35m10.836081578s) [upload=115928]: Done uploading segments... Mar 10 18:35:13.239351234 (103h35m10.836122261s) [upload=115928]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.30408602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 763.679205ms trace: https://jaeger.dev.storj.io/trace/62ea03c3a72aaf43 FAIL (2026-03-10 18:40:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 763.719962ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..X....................] duration: 2.506557849s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:39:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.50663063s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 483.500327ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:39:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.524563ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.133136338s trace: https://jaeger.dev.storj.io/trace/14d851485ec55b94 FAIL (2026-03-10 18:39:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== egments... Mar 10 18:34:13.234105804 (103h34m10.830876822s) [upload=115928]: Got next segment. Inline: false Mar 10 18:34:13.234171800 (103h34m10.830942823s) [upload=115928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:34:13.234177690 (103h34m10.830948712s) [upload=115928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:34:13.264565602 (103h34m10.861336626s) [seg_pos={0 0}, upload=115928]: Begin upload segment... Mar 10 18:34:13.264650554 (103h34m10.861421568s) [seg_pos={0 0}, upload=115928]: Done begin upload segment. Mar 10 18:34:13.264710110 (103h34m10.861481134s) [noise=true , address=45.91.116.9:4280 , piece_id=O6XIUDM5QEPO2S3TSGBRWFJOA6YNOPISRIKOKJMYIYVCY3W3O6FA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.265998754 (103h34m10.862769779s) [noise=true , address=77.90.31.14:4669 , piece_id=P7P7VFJ2CX2EXILR5AVF6ZQJSRWJ7HFBWB6APJUKQOETOPO7N3CQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.266563685 (103h34m10.863334713s) [noise=true , address=77.90.31.19:4863 , piece_id=DWBZ64GZUGWORUXIZGZVHTWA3VAB4GILLMMXVU5KB2VBGRX7FN2Q, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.266980797 (103h34m10.863751825s) [noise=true , address=193.149.164.68:4044 , piece_id=HBSJYACUD45I5CQBQ3PJZOWN4MBTXJCTGHMKHOMTUG5YCDISGPOA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267514292 (103h34m10.864285328s) [noise=true , address=79.127.205.235:4262 , piece_id=X6XAWZ64BAJP2EYT4PXLQHP6NTPCMRXSGHPTDF55V4Z3XXUZVO7A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267576292 (103h34m10.864347316s) [noise=true , address=193.149.164.69:4071 , piece_id=WVYSHF5Z35Z4DFCEVWVBKRRDLELXAV5URWMSFHNKCYCBQBKXBXIA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.267944808 (103h34m10.864715836s) [noise=true , address=193.149.164.67:4032 , piece_id=OYVBK5VQX2ICUTG6LTGB5DOEUKWA2Q42ANILPBUA3ELWA5XZ7OSQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268291059 (103h34m10.865062093s) [noise=true , address=79.127.205.234:4190 , piece_id=2DCWY6MVVYBU7R5R2G6LABDDL6YCBCYEQI3B5GL2P67SFCFU7ECQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268635689 (103h34m10.865406733s) [noise=true , address=79.127.205.237:4241 , piece_id=EMX5GFPZX3VLRD4AG3SDATNVA3YSJ6EAB4WXY3LTFE6AWEO7XZ4A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.268857349 (103h34m10.865628383s) [noise=true , address=77.90.31.11:4587 , piece_id=XDEV3CN5NW3VMYC7B3HYF2YLSQ6GV334LG2YYD2X4PIJQE3ZNEYA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.269379163 (103h34m10.866150189s) [noise=true , address=145.79.227.31:4822 , piece_id=7A2BFMPYL6EWUQC5RFZY6IIVXIVQLML6TSZVUP7VU4NIQLWYB63Q, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.269442067 (103h34m10.866213117s) [noise=true , address=77.90.31.13:4647 , piece_id=5DBAZJVJSJZF2AHFPAQBVTDKTF4LRXTJRNNLMSNDHD6G5TY366GQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.270023291 (103h34m10.866794323s) [noise=true , address=193.149.164.71:4298 , piece_id=VMRPXXIJIKZAWY6SCIA3FMAFIQ4XYI4LWXLVFRNMSKY5B6MV5DMQ, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.270558766 (103h34m10.867329801s) [noise=true , address=79.127.219.41:4954 , piece_id=UMD2OKNI36MLUAC5TOF6PVK573GN3JJSM3HXCYOLG25EGRPDVHJA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271066172 (103h34m10.867837205s) [noise=true , address=77.90.31.18:4837 , piece_id=N5RMIIIGCAQES6G6YXHCUX2YRVKIE764F57XQFGWOCKYOBJV33ZA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271143045 (103h34m10.867914074s) [noise=true , address=167.17.72.68:4645 , piece_id=76HO76ZYEW7YE2XGOZY3EJJHTZFW24B7IFGF3IMUV6IMYKTHLG2A, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.271561296 (103h34m10.86833233s) [noise=true , address=167.17.72.70:4755 , piece_id=NXGZWBH2UNVNSE7B2BHVBEC434N7MWHDYTDECVTZNB3O6GYPZUNA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.272067991 (103h34m10.868839022s) [noise=true , address=79.127.219.45:4878 , piece_id=GTM5BKXU6TQKMBCF5FOW34OKWVQLUJUJ4WINDYBZFR3ORI2OSHSA, seg_pos={0 0}, upload=115928]: Uploading piece... Mar 10 18:34:13.341870072 (103h34m10.938641097s) [upload=115928]: Next returned nil segment Mar 10 18:34:13.341887354 (103h34m10.938658381s) [upload=115928]: Waiting for error group managing segments... Mar 10 18:34:13.357317608 (103h34m10.954088641s) [noise=true , address=79.127.219.41:4954 , piece_id=UMD2OKNI36MLUAC5TOF6PVK573GN3JJSM3HXCYOLG25EGRPDVHJA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.369079638 (103h34m10.965850664s) [noise=true , address=45.91.116.9:4280 , piece_id=O6XIUDM5QEPO2S3TSGBRWFJOA6YNOPISRIKOKJMYIYVCY3W3O6FA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.383782380 (103h34m10.980553407s) [noise=true , address=145.79.227.31:4822 , piece_id=7A2BFMPYL6EWUQC5RFZY6IIVXIVQLML6TSZVUP7VU4NIQLWYB63Q, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.383844707 (103h34m10.980615732s) [noise=true , address=79.127.219.45:4878 , piece_id=GTM5BKXU6TQKMBCF5FOW34OKWVQLUJUJ4WINDYBZFR3ORI2OSHSA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.596292826 (103h34m11.19306384s) [noise=true , address=167.17.72.68:4645 , piece_id=76HO76ZYEW7YE2XGOZY3EJJHTZFW24B7IFGF3IMUV6IMYKTHLG2A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.657877077 (103h34m11.254648103s) [noise=true , address=167.17.72.70:4755 , piece_id=NXGZWBH2UNVNSE7B2BHVBEC434N7MWHDYTDECVTZNB3O6GYPZUNA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.815668472 (103h34m11.412439501s) [noise=true , address=79.127.205.234:4190 , piece_id=2DCWY6MVVYBU7R5R2G6LABDDL6YCBCYEQI3B5GL2P67SFCFU7ECQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.816829420 (103h34m11.413600446s) [noise=true , address=79.127.205.237:4241 , piece_id=EMX5GFPZX3VLRD4AG3SDATNVA3YSJ6EAB4WXY3LTFE6AWEO7XZ4A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.818063674 (103h34m11.414834697s) [noise=true , address=79.127.205.235:4262 , piece_id=X6XAWZ64BAJP2EYT4PXLQHP6NTPCMRXSGHPTDF55V4Z3XXUZVO7A, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.844922561 (103h34m11.441693585s) [noise=true , address=193.149.164.67:4032 , piece_id=OYVBK5VQX2ICUTG6LTGB5DOEUKWA2Q42ANILPBUA3ELWA5XZ7OSQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.862417635 (103h34m11.459188668s) [noise=true , address=193.149.164.71:4298 , piece_id=VMRPXXIJIKZAWY6SCIA3FMAFIQ4XYI4LWXLVFRNMSKY5B6MV5DMQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.864932308 (103h34m11.461703323s) [noise=true , address=193.149.164.69:4071 , piece_id=WVYSHF5Z35Z4DFCEVWVBKRRDLELXAV5URWMSFHNKCYCBQBKXBXIA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:34:13.874686150 (103h34m11.471457173s) [noise=true , address=193.149.164.68:4044 , piece_id=HBSJYACUD45I5CQBQ3PJZOWN4MBTXJCTGHMKHOMTUG5YCDISGPOA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: <nil> Mar 10 18:35:13.203204147 (103h35m10.799975179s) [noise=true , address=77.90.31.19:4863 , piece_id=DWBZ64GZUGWORUXIZGZVHTWA3VAB4GILLMMXVU5KB2VBGRX7FN2Q, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: rpc: dial tcp 77.90.31.19:4863: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203494517 (103h35m10.800265546s) [noise=true , address=77.90.31.18:4837 , piece_id=N5RMIIIGCAQES6G6YXHCUX2YRVKIE764F57XQFGWOCKYOBJV33ZA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: write tcp 159.223.143.39:56048->77.90.31.18:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203740816 (103h35m10.800511848s) [noise=true , address=77.90.31.13:4647 , piece_id=5DBAZJVJSJZF2AHFPAQBVTDKTF4LRXTJRNNLMSNDHD6G5TY366GQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: rpc: dial tcp 77.90.31.13:4647: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.203754450 (103h35m10.800525479s) [noise=true , address=77.90.31.11:4587 , piece_id=XDEV3CN5NW3VMYC7B3HYF2YLSQ6GV334LG2YYD2X4PIJQE3ZNEYA, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: rpc: dial tcp 77.90.31.11:4587: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.204001069 (103h35m10.800772099s) [noise=true , address=77.90.31.14:4669 , piece_id=P7P7VFJ2CX2EXILR5AVF6ZQJSRWJ7HFBWB6APJUKQOETOPO7N3CQ, seg_pos={0 0}, upload=115928]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:35:13.204030719 (103h35m10.800801732s) [seg_pos={0 0}, upload=115928]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:35:13.239310552 (103h35m10.836081578s) [upload=115928]: Done uploading segments... Mar 10 18:35:13.239351234 (103h35m10.836122261s) [upload=115928]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.133166352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 672.712028ms trace: https://jaeger.dev.storj.io/trace/62ea03c3a72aaf43 FAIL (2026-03-10 18:39:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 672.816389ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..X...................] duration: 2.558513947s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:39:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.558544746s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 669.885086ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:39:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 669.905878ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..X..................] duration: 2.144772784s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:38:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.144794533s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 796.544671ms trace: https://jaeger.dev.storj.io/trace/62ea03c3a72aaf43 FAIL (2026-03-10 18:38:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 796.583202ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)