Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 320364 reports (Page 398 of 6408)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 639.995934ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:54:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 640.021015ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 673.9911ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:54:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 674.019066ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..................X.............] duration: 628.199438ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:54:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 628.245318ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X...X...X........] duration: 1.20984029s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:53:53 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.209989696s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [................X..X........................................] duration: 4.093472585s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:53:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 4.093503315s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 496.650108ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:53:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.676551ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X..................X............] duration: 604.255661ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:53:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 604.285426ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.................X..X.......................................] duration: 4.912229175s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:53:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 4.912257346s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.......X.............] duration: 1.094777232s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:53:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.094807011s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 481.658655ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:53:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 481.682725ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..................X...........] duration: 575.964565ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:53:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 576.041623ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X...X...X.......] duration: 1.209453907s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:53:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.209639035s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 747.337589ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:52:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 747.505365ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X...X...X......] duration: 1.258865112s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:52:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.258892426s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.......X............] duration: 1.230283961s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:52:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.230312493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..................X..........] duration: 554.705262ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:52:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 554.733699ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................X..X......................................] duration: 4.302018278s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:52:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 4.302046929s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 705.229728ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:51:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 705.274714ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................X..X.....................................] duration: 3.422127178s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:51:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 3.422187469s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X...X...X.....] duration: 1.222059451s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:51:38 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.222075247s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.......X...........] duration: 1.269074047s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:51:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.269098737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..................X.........] duration: 644.025349ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:51:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 644.113678ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 472.866837ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:51:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.913109ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................X..X....................................] duration: 4.306471102s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:51:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 4.306528681s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 481.275296ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:50:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 481.349503ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X...X...X....] duration: 1.298777196s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:50:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.298808493s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.......X..........] duration: 1.252239347s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:50:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.252275231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..X...................................] duration: 13.514976864s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:50:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 13.515006828s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X..................X........] duration: 622.696735ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:50:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 622.7288ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 524.942028ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:49:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.013787ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.......X.........] duration: 1.364477779s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:49:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.364522221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X...X...X...] duration: 1.284464433s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:49:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.284547214s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......................X..X..................................] duration: 2.565598238s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:49:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.565626309s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..................X.......] duration: 700.819382ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:49:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 700.842204ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.......X........] duration: 1.304812505s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:48:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.304841085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 784.442875ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:48:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 784.472587ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..X.................................] duration: 6.92492024s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:48:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.92495056s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..................X......] duration: 828.118659ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:48:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 828.142864ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X...X...X..] duration: 1.204438871s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:48:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.204466976s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 515.03313ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:48:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.099944ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.......X.......] duration: 1.547702469s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:48:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.547726471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X...X...X.] duration: 1.37844855s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:48:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.378491954s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X...X...X] duration: 1m0.031006751s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:48:13 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.031028801s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..................X.....] duration: 657.210284ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:48:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 657.237665ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........................X..X................................] duration: 7.652152398s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:47:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 7.652258393s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 502.782281ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:47:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 502.829954ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..X...............................] duration: 12.657449289s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:47:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 12.657476059s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 494.479165ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:47:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.502314ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration: 1.259611931s trace: https://jaeger.dev.storj.io/trace/48f4a4942d823d32 FAIL (2026-03-10 18:47:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 03h42m15.405228152s) [upload=116080]: Got next segment. Inline: false Mar 10 18:42:17.810223679 (103h42m15.406994705s) [upload=116080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:42:17.810236964 (103h42m15.407007992s) [upload=116080]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:42:17.840222275 (103h42m15.436993287s) [seg_pos={0 0}, upload=116080]: Begin upload segment... Mar 10 18:42:17.840289868 (103h42m15.437060881s) [seg_pos={0 0}, upload=116080]: Done begin upload segment. Mar 10 18:42:17.840322464 (103h42m15.437093495s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.843898968 (103h42m15.440670001s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.844770315 (103h42m15.441541347s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845336789 (103h42m15.442107819s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.845789347 (103h42m15.442560361s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846158250 (103h42m15.442929274s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846523744 (103h42m15.443294776s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.846892520 (103h42m15.443663546s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847331251 (103h42m15.444102277s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.847751584 (103h42m15.444522607s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.850269395 (103h42m15.447040431s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.851909126 (103h42m15.448680159s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.853567965 (103h42m15.450338992s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.855073101 (103h42m15.45184413s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.856655425 (103h42m15.45342646s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.858432334 (103h42m15.455203367s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861168590 (103h42m15.457939622s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.861992744 (103h42m15.458763774s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Uploading piece... Mar 10 18:42:17.903321061 (103h42m15.500092083s) [upload=116080]: Next returned nil segment Mar 10 18:42:17.903338314 (103h42m15.500109329s) [upload=116080]: Waiting for error group managing segments... Mar 10 18:42:17.908477030 (103h42m15.505248066s) [noise=true , address=145.79.227.7:4322 , piece_id=PI4SGGIP34E5WD5W5E4MD42QYYKHPRT6SDWVDM34ZJPTYXYY54HA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.916315733 (103h42m15.513086757s) [noise=true , address=45.91.116.9:4269 , piece_id=UISDBPU2QROGCB7RSBHTYHIXUCXSZGXYBJIAX7LD4ZPGICNXYSXA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929093808 (103h42m15.525864826s) [noise=true , address=145.79.227.18:4035 , piece_id=UY7VEVKCUNAIPAXUBGX3EYDVTJQAQR2HFP5A4BPGJHKU43YM647Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:17.929915992 (103h42m15.526687022s) [noise=true , address=145.79.227.24:4600 , piece_id=GBGRIANXT7CGBK4DNKOYUN4RPLO4MLHLRP6MYYIPLQIEZGPLF7GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.233365236 (103h42m15.830136253s) [noise=true , address=167.17.72.70:4772 , piece_id=HELLMCKNLPGI45BRAMSUBEYO3M3OD3OO4VUFSKT4O23TAWHA7X3Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.326771985 (103h42m15.923543028s) [noise=true , address=145.79.227.11:4758 , piece_id=75KSKKNNBANBUU4HFPSTNQJTCCEYSXCAIGTGE7NCEPXBEU3WM2GA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.388920554 (103h42m15.98569158s) [noise=true , address=79.127.205.233:4170 , piece_id=UFUOBCOEWJBJTJQ6DG7VPZTPBMLXYLOHFZRTFAJQRLE3MCMAUUFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.398546783 (103h42m15.995317823s) [noise=true , address=79.127.205.237:4238 , piece_id=SH7XVKENFY5JINKCUXY2Q7G344FBABGYTARR6OH2CIZS2L24PKSQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.422892737 (103h42m16.019663767s) [noise=true , address=193.149.164.71:4318 , piece_id=RDERJITLK2BCMDVC64GSKTUIK3YB66NJIG3P3RC4PH74MAPSLV7Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.449554006 (103h42m16.046325035s) [noise=true , address=193.149.164.70:4106 , piece_id=Z4OGLOSOPP23JJNT2J4E4AUBUBYIQ22ABBXJZCR4KSG4WXUR7OFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.470722723 (103h42m16.067493759s) [noise=true , address=193.149.164.68:4057 , piece_id=74OCR3OTZD4YSUWVTWL3JXXH72IAORIN3FU4G6F5O2TQWU6UCMEA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.522634796 (103h42m16.11940582s) [noise=true , address=193.149.164.67:4019 , piece_id=CBT45NPKHNCXAZEVIPP3B6JZHHXGQUCLUTYICZG744JL26SWC63Q, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:42:18.654055854 (103h42m16.250826878s) [noise=true , address=167.17.72.68:4664 , piece_id=JWK4FAZ2I4GC4V4TBN4RZZD7XO6IEQ3CR5TTMYLSSVRXPO6SJMGQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: <nil> Mar 10 18:43:17.776142471 (103h43m15.372913498s) [noise=true , address=77.90.31.19:4861 , piece_id=5SR7Z6472ZIQHTDYPPNQMQFS4BXYRVIWR65J2LTU6BCRFDBGPHTQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: write tcp 159.223.143.39:49960->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776318420 (103h43m15.373089455s) [noise=true , address=77.90.31.12:4607 , piece_id=MJOGOCK4JDW2Q5N7J7X5IZHXZGEH7ZPY6MK2ZLZBYNNHV7YEQYZA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, address:77.90.31.12:4607): protocol: rpc: dial tcp 77.90.31.12:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776631481 (103h43m15.373402513s) [noise=true , address=77.90.31.18:4832 , piece_id=ILTUXE4N2GNCQF2NQAHQ6IS4T2H64ULIFHYEAI4MFZVKICC3DPNA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; context deadline exceeded Mar 10 18:43:17.776850169 (103h43m15.373621203s) [noise=true , address=77.90.31.10:4524 , piece_id=CJ5VEN7DXUFTCWILXCZ2FSMFOCHRQ3TPAVTBU324ZYMM3TTZADFQ, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:52866->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.776943845 (103h43m15.373714872s) [noise=true , address=77.90.31.15:4728 , piece_id=2L73KYRU2I5RIAI34HMCQQ5YJ3WNENBIW3AUNGS74Q4A7I4PWWVA, seg_pos={0 0}, upload=116080]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: write tcp 159.223.143.39:54478->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:43:17.777038168 (103h43m15.373809193s) [seg_pos={0 0}, upload=116080]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:43:17.802429117 (103h43m15.399200141s) [upload=116080]: Done uploading segments... Mar 10 18:43:17.802678296 (103h43m15.399449319s) [upload=116080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.259642086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..................X....] duration: 676.648742ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:47:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 677.266936ms core uplink us-select-1 us1-placement-20

1
...
396
397
398
399
400
...
6408