Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 716.276532ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 19:02:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 716.330357ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X...X...X..................] duration: 1.420404123s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 19:02:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.420441469s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 552.754614ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 19:01:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 552.794058ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X..................X.......................] duration: 643.418594ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:01:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 643.446179ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X.......X.............X....X....] duration: 1.342445213s trace: https://jaeger.dev.storj.io/trace/5551066fb24b6624 FAIL (2026-03-10 19:01:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=116365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:57:27.484826534 (103h57m25.081597557s) [upload=116365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:57:27.521721468 (103h57m25.11849248s) [seg_pos={0 0}, upload=116365]: Begin upload segment... Mar 10 18:57:27.521868020 (103h57m25.118639046s) [seg_pos={0 0}, upload=116365]: Done begin upload segment. Mar 10 18:57:27.521902956 (103h57m25.118673979s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.521983909 (103h57m25.118754933s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.523613945 (103h57m25.120384968s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524285299 (103h57m25.121056331s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524949601 (103h57m25.121720624s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525484827 (103h57m25.12225585s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525969502 (103h57m25.122740528s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.526481271 (103h57m25.123252294s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527363350 (103h57m25.124134383s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527903397 (103h57m25.12467443s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.528459198 (103h57m25.125230242s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529239126 (103h57m25.12601015s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529855918 (103h57m25.126626944s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.530483568 (103h57m25.1272546s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531044521 (103h57m25.127815545s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531608342 (103h57m25.128379366s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532233632 (103h57m25.12900466s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532810194 (103h57m25.129581217s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.598647582 (103h57m25.195418592s) [upload=116365]: Next returned nil segment Mar 10 18:57:27.598660671 (103h57m25.195431693s) [upload=116365]: Waiting for error group managing segments... Mar 10 18:57:27.617260011 (103h57m25.214031035s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623564891 (103h57m25.22033592s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623817360 (103h57m25.220588374s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.624221744 (103h57m25.220992769s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.627462959 (103h57m25.224233983s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.654127655 (103h57m25.250898681s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.900823571 (103h57m25.497594593s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.063905047 (103h57m25.660676071s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.073501018 (103h57m25.670272043s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.113442671 (103h57m25.710213697s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.120035234 (103h57m25.716806258s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.191954998 (103h57m25.788726025s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:58:27.450793500 (103h58m25.047564535s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.450883537 (103h58m25.04765457s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: rpc: dial tcp 77.90.31.21:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451074703 (103h58m25.047845735s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: rpc: dial tcp 77.90.31.12:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451150309 (103h58m25.047921346s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: rpc: dial tcp 77.90.31.14:4672: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451331738 (103h58m25.048102771s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:58:27.451342106 (103h58m25.048113141s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: write tcp 159.223.143.39:37834->77.90.31.16:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451368018 (103h58m25.048139029s) [seg_pos={0 0}, upload=116365]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:58:27.484713412 (103h58m25.08148443s) [upload=116365]: Done uploading segments... Mar 10 18:58:27.484792918 (103h58m25.081563946s) [upload=116365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.342469778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [......X..X................................................X.] duration: 2.600373938s trace: https://jaeger.dev.storj.io/trace/41fc19646dedf2b FAIL (2026-03-10 19:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 59:48.871752042 (103h59m46.468523065s) [noise=true , address=89.149.201.147:4019 , piece_id=LX4563HAFDFGRKQRAHMUQW73C53H6IPUGSTIIYZZHCGVI5BOGM7Q, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.871766366 (103h59m46.468537394s) [noise=true , address=185.167.204.205:4028 , piece_id=XLWWPIGHAJRGX5IHVEFKCOU2FTEBPNIJHJG5VB7ZWTA4MV6NGESA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.872236431 (103h59m46.469007465s) [noise=true , address=81.171.14.70:4041 , piece_id=X3ZIMMXALIAS344XK4AVU3JUTIOX6ALV3H37ME5D4YUXQSJFC72Q, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.872476023 (103h59m46.469247056s) [noise=true , address=89.149.201.147:4003 , piece_id=AY3EGMTSG27LE5UPTJDEU2U2LIZP2KGHKWEPEPIWRSROK6N3IPWA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.872642319 (103h59m46.469413365s) [noise=true , address=185.167.204.206:4049 , piece_id=YTITX2QWUAY5V6QPOJM2K2BSG72DI2I7TA7QUMCFGC2OVRSM6TBQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873017409 (103h59m46.469788442s) [noise=true , address=81.171.14.70:4039 , piece_id=2CMRZ7PXFO4KRMGGWDGL5WMM3DZDS5CRMN4J52UIRJQ2U3ADY2XA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873112445 (103h59m46.469883471s) [noise=true , address=81.171.14.70:4028 , piece_id=2IBQ6E43VW2EHJNXLYPBQNRPSUVLHCHBI3QDSXUDGYCWPVYK4Q3A, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873504435 (103h59m46.470275458s) [noise=true , address=185.167.204.206:4053 , piece_id=PA33DAERI5CQ3D5OFWQDMCINLDXZKS4CEIUT22APHR4ZWYSNW7UA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873518754 (103h59m46.470289784s) [noise=true , address=81.171.14.70:4027 , piece_id=GAHTAO4BV2ZLDTVQWFRIUBFU62JHRJKDQUZRYMZ5PVHBKC2H7TUQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873935290 (103h59m46.470706313s) [noise=true , address=89.149.201.147:4011 , piece_id=MHGBZPAZHHTNFWDX4WJNLIWEBEZODIK3EHP6AOKMCC7XUI2DF4MQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873941072 (103h59m46.470712101s) [noise=true , address=185.167.204.206:4061 , piece_id=TR7JQ2VQNGAB7XPT74UJ73NKL34GFLCUNEQLSX23PN2HUSIATCRQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.878089390 (103h59m46.474860423s) [noise=true , address=185.167.204.205:4005 , piece_id=S3OUWBUADHJFP2IESZATQF66EUIKRNXTLKGYPWNELN6FPQY7AEDA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:49.248411606 (103h59m46.84518263s) [upload=116406]: Next returned nil segment Mar 10 18:59:49.248425455 (103h59m46.845196479s) [upload=116406]: Waiting for error group managing segments... Mar 10 18:59:49.474071618 (103h59m47.070842642s) [noise=true , address=81.171.14.70:4039 , piece_id=2CMRZ7PXFO4KRMGGWDGL5WMM3DZDS5CRMN4J52UIRJQ2U3ADY2XA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.480292316 (103h59m47.07706334s) [noise=true , address=81.171.14.70:4028 , piece_id=2IBQ6E43VW2EHJNXLYPBQNRPSUVLHCHBI3QDSXUDGYCWPVYK4Q3A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.489340334 (103h59m47.086111357s) [noise=true , address=81.171.14.70:4027 , piece_id=GAHTAO4BV2ZLDTVQWFRIUBFU62JHRJKDQUZRYMZ5PVHBKC2H7TUQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.504000682 (103h59m47.100771706s) [noise=true , address=81.171.14.70:4025 , piece_id=D34FMTR4XLUI6HWNTNBC6ESS42BC33F4WYA4ABR2CHJJ6H5S7YIQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.532473713 (103h59m47.129244742s) [noise=true , address=89.149.201.147:4019 , piece_id=LX4563HAFDFGRKQRAHMUQW73C53H6IPUGSTIIYZZHCGVI5BOGM7Q, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.537613891 (103h59m47.134384904s) [noise=true , address=89.149.201.147:4011 , piece_id=MHGBZPAZHHTNFWDX4WJNLIWEBEZODIK3EHP6AOKMCC7XUI2DF4MQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.549489567 (103h59m47.146260589s) [noise=true , address=89.149.201.147:4014 , piece_id=5Q7HBKB3WLFUKAGWJ64IXNCXPAATDIFUCYN6N2V2XSDH2VWBLWJA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.549783369 (103h59m47.146554395s) [noise=true , address=89.149.201.147:4010 , piece_id=WRP553BFHMTHKDA52R4B27FRRYDITSUXJMWZIQGETJQ534RXOSOA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.550944998 (103h59m47.147716022s) [noise=true , address=81.171.14.70:4041 , piece_id=X3ZIMMXALIAS344XK4AVU3JUTIOX6ALV3H37ME5D4YUXQSJFC72Q, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.552358477 (103h59m47.1491295s) [noise=true , address=185.167.204.206:4066 , piece_id=LY33PR2OIW6UHZDFP453BGMG6IRJEYTTU7WD3JFPEFLQP6ERGUOA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.556323257 (103h59m47.153094271s) [noise=true , address=185.167.204.206:4061 , piece_id=TR7JQ2VQNGAB7XPT74UJ73NKL34GFLCUNEQLSX23PN2HUSIATCRQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.557938128 (103h59m47.154709158s) [noise=true , address=185.167.204.205:4030 , piece_id=FMVAQDEEBV2O4IVQDOLXUKF3DSDMDBM7J2WAHQ5LL3XIQDEXDXAA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.558180752 (103h59m47.154951776s) [noise=true , address=185.167.204.205:4005 , piece_id=S3OUWBUADHJFP2IESZATQF66EUIKRNXTLKGYPWNELN6FPQY7AEDA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.558651610 (103h59m47.155422634s) [noise=true , address=89.149.201.147:4009 , piece_id=6DUG4IKPT57NLKPPLQYBMBHICPG65BHRZCTTIGFJZJ5253XU42BA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.558678943 (103h59m47.155449966s) [noise=true , address=89.149.201.147:4023 , piece_id=2SU7S4JGCLGDYZUJAP7R7M4NNEMLHTRNLZQV2APBOGBK3WDKDYZQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.575712930 (103h59m47.172483953s) [noise=true , address=185.167.204.206:4060 , piece_id=G6JD5YY4L3BVJADVWRRKCJFBFKR7LK2XL2YSVCMFUBCRALB2G2AQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.593797789 (103h59m47.190568818s) [noise=true , address=185.167.204.206:4049 , piece_id=YTITX2QWUAY5V6QPOJM2K2BSG72DI2I7TA7QUMCFGC2OVRSM6TBQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.598510090 (103h59m47.195281116s) [noise=true , address=89.149.201.147:4012 , piece_id=OT7VVAXSBT262LDOIQPT4LGSBAGQ7ICJEVBGXM7UYUINPES5DI2A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.641662861 (103h59m47.238433876s) [noise=true , address=185.167.204.206:4054 , piece_id=OMEHHTI27QJPSX4YA6ZF4EQR22JY2XGU6FHLI65P5YNXVOQGAL3A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:50.048638281 (103h59m47.645409304s) [noise=true , address=89.149.201.147:4003 , piece_id=AY3EGMTSG27LE5UPTJDEU2U2LIZP2KGHKWEPEPIWRSROK6N3IPWA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:50.109812159 (103h59m47.706583181s) [noise=true , address=185.167.204.205:4019 , piece_id=GCHTNKUHMMKWJ2QHDFLO5EHTWNWL33MXYBTM3WAWF773XBWZHX6A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:51.470994001 (103h59m49.067765025s) [noise=true , address=185.167.204.205:4028 , piece_id=XLWWPIGHAJRGX5IHVEFKCOU2FTEBPNIJHJG5VB7ZWTA4MV6NGESA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:52.215239364 (103h59m49.812010379s) [noise=true , address=185.167.204.206:4056 , piece_id=T4SSK3GQ32JLSSCBFTQGWNEB3SI6VGJBOQDIIJSBOHIVCDMEEZAQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:52.415825498 (103h59m50.012596522s) [noise=true , address=185.167.204.206:4063 , piece_id=NYAMWJ6LZSW5BZS3EZRO2HSOCICIEMSVLZ3FF3W764VSDDBJR7KQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:53.171677840 (103h59m50.768448864s) [noise=true , address=185.167.204.206:4053 , piece_id=PA33DAERI5CQ3D5OFWQDMCINLDXZKS4CEIUT22APHR4ZWYSNW7UA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 19:00:48.501538387 (104h0m46.09830942s) [noise=true , address=89.149.201.147:4001 , piece_id=YLIRWGW7D6LGIDT5Q475DY7LZDSLNP5DB6UETRFQ6GZALNBQTDWQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: read tcp 159.223.143.39:54340->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:00:48.501665952 (104h0m46.09843697s) [noise=true , address=89.149.201.147:4007 , piece_id=ZNB53UYB4VYMTUBTT2YEOSAXDZVTEATKM5YZGQNS6SWDNFUBMQFQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:33044->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:00:48.501953179 (104h0m46.09872421s) [noise=true , address=89.149.201.147:4006 , piece_id=4UFBWFA5SK7BKB6EGF6HW6MSQPKKMJKXZDHW6UNDS3I6TP3TLFYA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: ecclient: upload failed (node:12YTf9jvAKd4gVTxTZC6gGLdXC17ygbrqvqoKtmPbtHyc65YjDU, address:89.149.201.147:4006): protocol: read tcp 159.223.143.39:60806->89.149.201.147:4006: use of closed network connection; context deadline exceeded Mar 10 19:00:48.501990253 (104h0m46.09876127s) [seg_pos={0 0}, upload=116406]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:00:48.600564516 (104h0m46.197335537s) [upload=116406]: Done uploading segments... Mar 10 19:00:48.600611284 (104h0m46.197382321s) [upload=116406]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.60040449s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X...X...X.................] duration: 1.09823385s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 19:00:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.098279793s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.......X..X................................................X] duration: 1m0.102946961s trace: https://jaeger.dev.storj.io/trace/41fc19646dedf2b FAIL (2026-03-10 19:00:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 59:48.871752042 (103h59m46.468523065s) [noise=true , address=89.149.201.147:4019 , piece_id=LX4563HAFDFGRKQRAHMUQW73C53H6IPUGSTIIYZZHCGVI5BOGM7Q, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.871766366 (103h59m46.468537394s) [noise=true , address=185.167.204.205:4028 , piece_id=XLWWPIGHAJRGX5IHVEFKCOU2FTEBPNIJHJG5VB7ZWTA4MV6NGESA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.872236431 (103h59m46.469007465s) [noise=true , address=81.171.14.70:4041 , piece_id=X3ZIMMXALIAS344XK4AVU3JUTIOX6ALV3H37ME5D4YUXQSJFC72Q, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.872476023 (103h59m46.469247056s) [noise=true , address=89.149.201.147:4003 , piece_id=AY3EGMTSG27LE5UPTJDEU2U2LIZP2KGHKWEPEPIWRSROK6N3IPWA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.872642319 (103h59m46.469413365s) [noise=true , address=185.167.204.206:4049 , piece_id=YTITX2QWUAY5V6QPOJM2K2BSG72DI2I7TA7QUMCFGC2OVRSM6TBQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873017409 (103h59m46.469788442s) [noise=true , address=81.171.14.70:4039 , piece_id=2CMRZ7PXFO4KRMGGWDGL5WMM3DZDS5CRMN4J52UIRJQ2U3ADY2XA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873112445 (103h59m46.469883471s) [noise=true , address=81.171.14.70:4028 , piece_id=2IBQ6E43VW2EHJNXLYPBQNRPSUVLHCHBI3QDSXUDGYCWPVYK4Q3A, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873504435 (103h59m46.470275458s) [noise=true , address=185.167.204.206:4053 , piece_id=PA33DAERI5CQ3D5OFWQDMCINLDXZKS4CEIUT22APHR4ZWYSNW7UA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873518754 (103h59m46.470289784s) [noise=true , address=81.171.14.70:4027 , piece_id=GAHTAO4BV2ZLDTVQWFRIUBFU62JHRJKDQUZRYMZ5PVHBKC2H7TUQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873935290 (103h59m46.470706313s) [noise=true , address=89.149.201.147:4011 , piece_id=MHGBZPAZHHTNFWDX4WJNLIWEBEZODIK3EHP6AOKMCC7XUI2DF4MQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.873941072 (103h59m46.470712101s) [noise=true , address=185.167.204.206:4061 , piece_id=TR7JQ2VQNGAB7XPT74UJ73NKL34GFLCUNEQLSX23PN2HUSIATCRQ, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:48.878089390 (103h59m46.474860423s) [noise=true , address=185.167.204.205:4005 , piece_id=S3OUWBUADHJFP2IESZATQF66EUIKRNXTLKGYPWNELN6FPQY7AEDA, seg_pos={0 0}, upload=116406]: Uploading piece... Mar 10 18:59:49.248411606 (103h59m46.84518263s) [upload=116406]: Next returned nil segment Mar 10 18:59:49.248425455 (103h59m46.845196479s) [upload=116406]: Waiting for error group managing segments... Mar 10 18:59:49.474071618 (103h59m47.070842642s) [noise=true , address=81.171.14.70:4039 , piece_id=2CMRZ7PXFO4KRMGGWDGL5WMM3DZDS5CRMN4J52UIRJQ2U3ADY2XA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.480292316 (103h59m47.07706334s) [noise=true , address=81.171.14.70:4028 , piece_id=2IBQ6E43VW2EHJNXLYPBQNRPSUVLHCHBI3QDSXUDGYCWPVYK4Q3A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.489340334 (103h59m47.086111357s) [noise=true , address=81.171.14.70:4027 , piece_id=GAHTAO4BV2ZLDTVQWFRIUBFU62JHRJKDQUZRYMZ5PVHBKC2H7TUQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.504000682 (103h59m47.100771706s) [noise=true , address=81.171.14.70:4025 , piece_id=D34FMTR4XLUI6HWNTNBC6ESS42BC33F4WYA4ABR2CHJJ6H5S7YIQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.532473713 (103h59m47.129244742s) [noise=true , address=89.149.201.147:4019 , piece_id=LX4563HAFDFGRKQRAHMUQW73C53H6IPUGSTIIYZZHCGVI5BOGM7Q, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.537613891 (103h59m47.134384904s) [noise=true , address=89.149.201.147:4011 , piece_id=MHGBZPAZHHTNFWDX4WJNLIWEBEZODIK3EHP6AOKMCC7XUI2DF4MQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.549489567 (103h59m47.146260589s) [noise=true , address=89.149.201.147:4014 , piece_id=5Q7HBKB3WLFUKAGWJ64IXNCXPAATDIFUCYN6N2V2XSDH2VWBLWJA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.549783369 (103h59m47.146554395s) [noise=true , address=89.149.201.147:4010 , piece_id=WRP553BFHMTHKDA52R4B27FRRYDITSUXJMWZIQGETJQ534RXOSOA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.550944998 (103h59m47.147716022s) [noise=true , address=81.171.14.70:4041 , piece_id=X3ZIMMXALIAS344XK4AVU3JUTIOX6ALV3H37ME5D4YUXQSJFC72Q, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.552358477 (103h59m47.1491295s) [noise=true , address=185.167.204.206:4066 , piece_id=LY33PR2OIW6UHZDFP453BGMG6IRJEYTTU7WD3JFPEFLQP6ERGUOA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.556323257 (103h59m47.153094271s) [noise=true , address=185.167.204.206:4061 , piece_id=TR7JQ2VQNGAB7XPT74UJ73NKL34GFLCUNEQLSX23PN2HUSIATCRQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.557938128 (103h59m47.154709158s) [noise=true , address=185.167.204.205:4030 , piece_id=FMVAQDEEBV2O4IVQDOLXUKF3DSDMDBM7J2WAHQ5LL3XIQDEXDXAA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.558180752 (103h59m47.154951776s) [noise=true , address=185.167.204.205:4005 , piece_id=S3OUWBUADHJFP2IESZATQF66EUIKRNXTLKGYPWNELN6FPQY7AEDA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.558651610 (103h59m47.155422634s) [noise=true , address=89.149.201.147:4009 , piece_id=6DUG4IKPT57NLKPPLQYBMBHICPG65BHRZCTTIGFJZJ5253XU42BA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.558678943 (103h59m47.155449966s) [noise=true , address=89.149.201.147:4023 , piece_id=2SU7S4JGCLGDYZUJAP7R7M4NNEMLHTRNLZQV2APBOGBK3WDKDYZQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.575712930 (103h59m47.172483953s) [noise=true , address=185.167.204.206:4060 , piece_id=G6JD5YY4L3BVJADVWRRKCJFBFKR7LK2XL2YSVCMFUBCRALB2G2AQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.593797789 (103h59m47.190568818s) [noise=true , address=185.167.204.206:4049 , piece_id=YTITX2QWUAY5V6QPOJM2K2BSG72DI2I7TA7QUMCFGC2OVRSM6TBQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.598510090 (103h59m47.195281116s) [noise=true , address=89.149.201.147:4012 , piece_id=OT7VVAXSBT262LDOIQPT4LGSBAGQ7ICJEVBGXM7UYUINPES5DI2A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:49.641662861 (103h59m47.238433876s) [noise=true , address=185.167.204.206:4054 , piece_id=OMEHHTI27QJPSX4YA6ZF4EQR22JY2XGU6FHLI65P5YNXVOQGAL3A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:50.048638281 (103h59m47.645409304s) [noise=true , address=89.149.201.147:4003 , piece_id=AY3EGMTSG27LE5UPTJDEU2U2LIZP2KGHKWEPEPIWRSROK6N3IPWA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:50.109812159 (103h59m47.706583181s) [noise=true , address=185.167.204.205:4019 , piece_id=GCHTNKUHMMKWJ2QHDFLO5EHTWNWL33MXYBTM3WAWF773XBWZHX6A, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:51.470994001 (103h59m49.067765025s) [noise=true , address=185.167.204.205:4028 , piece_id=XLWWPIGHAJRGX5IHVEFKCOU2FTEBPNIJHJG5VB7ZWTA4MV6NGESA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:52.215239364 (103h59m49.812010379s) [noise=true , address=185.167.204.206:4056 , piece_id=T4SSK3GQ32JLSSCBFTQGWNEB3SI6VGJBOQDIIJSBOHIVCDMEEZAQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:52.415825498 (103h59m50.012596522s) [noise=true , address=185.167.204.206:4063 , piece_id=NYAMWJ6LZSW5BZS3EZRO2HSOCICIEMSVLZ3FF3W764VSDDBJR7KQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 18:59:53.171677840 (103h59m50.768448864s) [noise=true , address=185.167.204.206:4053 , piece_id=PA33DAERI5CQ3D5OFWQDMCINLDXZKS4CEIUT22APHR4ZWYSNW7UA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: <nil> Mar 10 19:00:48.501538387 (104h0m46.09830942s) [noise=true , address=89.149.201.147:4001 , piece_id=YLIRWGW7D6LGIDT5Q475DY7LZDSLNP5DB6UETRFQ6GZALNBQTDWQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: ecclient: upload failed (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8, address:89.149.201.147:4001): protocol: read tcp 159.223.143.39:54340->89.149.201.147:4001: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:00:48.501665952 (104h0m46.09843697s) [noise=true , address=89.149.201.147:4007 , piece_id=ZNB53UYB4VYMTUBTT2YEOSAXDZVTEATKM5YZGQNS6SWDNFUBMQFQ, seg_pos={0 0}, upload=116406]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:33044->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:00:48.501953179 (104h0m46.09872421s) [noise=true , address=89.149.201.147:4006 , piece_id=4UFBWFA5SK7BKB6EGF6HW6MSQPKKMJKXZDHW6UNDS3I6TP3TLFYA, seg_pos={0 0}, upload=116406]: Done uploading piece. err: ecclient: upload failed (node:12YTf9jvAKd4gVTxTZC6gGLdXC17ygbrqvqoKtmPbtHyc65YjDU, address:89.149.201.147:4006): protocol: read tcp 159.223.143.39:60806->89.149.201.147:4006: use of closed network connection; context deadline exceeded Mar 10 19:00:48.501990253 (104h0m46.09876127s) [seg_pos={0 0}, upload=116406]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:00:48.600564516 (104h0m46.197335537s) [upload=116406]: Done uploading segments... Mar 10 19:00:48.600611284 (104h0m46.197382321s) [upload=116406]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 1m0.102971286s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X..................X......................] duration: 620.281306ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:00:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 620.312578ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 514.119633ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 19:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.204002ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X...X...X................] duration: 1.428542943s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 19:00:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.428590705s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.......X.............X....X...] duration: 1.263337303s trace: https://jaeger.dev.storj.io/trace/5551066fb24b6624 FAIL (2026-03-10 19:00:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=116365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:57:27.484826534 (103h57m25.081597557s) [upload=116365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:57:27.521721468 (103h57m25.11849248s) [seg_pos={0 0}, upload=116365]: Begin upload segment... Mar 10 18:57:27.521868020 (103h57m25.118639046s) [seg_pos={0 0}, upload=116365]: Done begin upload segment. Mar 10 18:57:27.521902956 (103h57m25.118673979s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.521983909 (103h57m25.118754933s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.523613945 (103h57m25.120384968s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524285299 (103h57m25.121056331s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524949601 (103h57m25.121720624s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525484827 (103h57m25.12225585s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525969502 (103h57m25.122740528s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.526481271 (103h57m25.123252294s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527363350 (103h57m25.124134383s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527903397 (103h57m25.12467443s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.528459198 (103h57m25.125230242s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529239126 (103h57m25.12601015s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529855918 (103h57m25.126626944s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.530483568 (103h57m25.1272546s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531044521 (103h57m25.127815545s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531608342 (103h57m25.128379366s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532233632 (103h57m25.12900466s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532810194 (103h57m25.129581217s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.598647582 (103h57m25.195418592s) [upload=116365]: Next returned nil segment Mar 10 18:57:27.598660671 (103h57m25.195431693s) [upload=116365]: Waiting for error group managing segments... Mar 10 18:57:27.617260011 (103h57m25.214031035s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623564891 (103h57m25.22033592s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623817360 (103h57m25.220588374s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.624221744 (103h57m25.220992769s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.627462959 (103h57m25.224233983s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.654127655 (103h57m25.250898681s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.900823571 (103h57m25.497594593s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.063905047 (103h57m25.660676071s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.073501018 (103h57m25.670272043s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.113442671 (103h57m25.710213697s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.120035234 (103h57m25.716806258s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.191954998 (103h57m25.788726025s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:58:27.450793500 (103h58m25.047564535s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.450883537 (103h58m25.04765457s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: rpc: dial tcp 77.90.31.21:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451074703 (103h58m25.047845735s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: rpc: dial tcp 77.90.31.12:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451150309 (103h58m25.047921346s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: rpc: dial tcp 77.90.31.14:4672: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451331738 (103h58m25.048102771s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:58:27.451342106 (103h58m25.048113141s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: write tcp 159.223.143.39:37834->77.90.31.16:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451368018 (103h58m25.048139029s) [seg_pos={0 0}, upload=116365]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:58:27.484713412 (103h58m25.08148443s) [upload=116365]: Done uploading segments... Mar 10 18:58:27.484792918 (103h58m25.081563946s) [upload=116365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.263368763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X.......X.............X....X..] duration: 1.244806298s trace: https://jaeger.dev.storj.io/trace/5551066fb24b6624 FAIL (2026-03-10 18:59:54 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=116365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:57:27.484826534 (103h57m25.081597557s) [upload=116365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:57:27.521721468 (103h57m25.11849248s) [seg_pos={0 0}, upload=116365]: Begin upload segment... Mar 10 18:57:27.521868020 (103h57m25.118639046s) [seg_pos={0 0}, upload=116365]: Done begin upload segment. Mar 10 18:57:27.521902956 (103h57m25.118673979s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.521983909 (103h57m25.118754933s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.523613945 (103h57m25.120384968s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524285299 (103h57m25.121056331s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524949601 (103h57m25.121720624s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525484827 (103h57m25.12225585s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525969502 (103h57m25.122740528s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.526481271 (103h57m25.123252294s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527363350 (103h57m25.124134383s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527903397 (103h57m25.12467443s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.528459198 (103h57m25.125230242s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529239126 (103h57m25.12601015s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529855918 (103h57m25.126626944s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.530483568 (103h57m25.1272546s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531044521 (103h57m25.127815545s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531608342 (103h57m25.128379366s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532233632 (103h57m25.12900466s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532810194 (103h57m25.129581217s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.598647582 (103h57m25.195418592s) [upload=116365]: Next returned nil segment Mar 10 18:57:27.598660671 (103h57m25.195431693s) [upload=116365]: Waiting for error group managing segments... Mar 10 18:57:27.617260011 (103h57m25.214031035s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623564891 (103h57m25.22033592s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623817360 (103h57m25.220588374s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.624221744 (103h57m25.220992769s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.627462959 (103h57m25.224233983s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.654127655 (103h57m25.250898681s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.900823571 (103h57m25.497594593s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.063905047 (103h57m25.660676071s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.073501018 (103h57m25.670272043s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.113442671 (103h57m25.710213697s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.120035234 (103h57m25.716806258s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.191954998 (103h57m25.788726025s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:58:27.450793500 (103h58m25.047564535s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.450883537 (103h58m25.04765457s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: rpc: dial tcp 77.90.31.21:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451074703 (103h58m25.047845735s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: rpc: dial tcp 77.90.31.12:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451150309 (103h58m25.047921346s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: rpc: dial tcp 77.90.31.14:4672: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451331738 (103h58m25.048102771s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:58:27.451342106 (103h58m25.048113141s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: write tcp 159.223.143.39:37834->77.90.31.16:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451368018 (103h58m25.048139029s) [seg_pos={0 0}, upload=116365]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:58:27.484713412 (103h58m25.08148443s) [upload=116365]: Done uploading segments... Mar 10 18:58:27.484792918 (103h58m25.081563946s) [upload=116365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.244838632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X..................X.....................] duration: 689.841464ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:59:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 689.868718ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X...X...X...............] duration: 1.315175598s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:59:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.315225714s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X..................X....................] duration: 914.815344ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:59:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 914.84284ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 686.418387ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:59:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 686.45722ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..................X...................] duration: 866.563396ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:59:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 866.622308ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........X..X................................................] duration: 7.472761724s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:58:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 7.472808995s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X...X...X..............] duration: 1.280026842s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:58:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.280066963s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X.......X.............X....X.] duration: 1.41577907s trace: https://jaeger.dev.storj.io/trace/5551066fb24b6624 FAIL (2026-03-10 18:58:43 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=116365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:57:27.484826534 (103h57m25.081597557s) [upload=116365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:57:27.521721468 (103h57m25.11849248s) [seg_pos={0 0}, upload=116365]: Begin upload segment... Mar 10 18:57:27.521868020 (103h57m25.118639046s) [seg_pos={0 0}, upload=116365]: Done begin upload segment. Mar 10 18:57:27.521902956 (103h57m25.118673979s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.521983909 (103h57m25.118754933s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.523613945 (103h57m25.120384968s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524285299 (103h57m25.121056331s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524949601 (103h57m25.121720624s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525484827 (103h57m25.12225585s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525969502 (103h57m25.122740528s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.526481271 (103h57m25.123252294s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527363350 (103h57m25.124134383s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527903397 (103h57m25.12467443s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.528459198 (103h57m25.125230242s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529239126 (103h57m25.12601015s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529855918 (103h57m25.126626944s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.530483568 (103h57m25.1272546s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531044521 (103h57m25.127815545s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531608342 (103h57m25.128379366s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532233632 (103h57m25.12900466s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532810194 (103h57m25.129581217s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.598647582 (103h57m25.195418592s) [upload=116365]: Next returned nil segment Mar 10 18:57:27.598660671 (103h57m25.195431693s) [upload=116365]: Waiting for error group managing segments... Mar 10 18:57:27.617260011 (103h57m25.214031035s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623564891 (103h57m25.22033592s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623817360 (103h57m25.220588374s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.624221744 (103h57m25.220992769s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.627462959 (103h57m25.224233983s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.654127655 (103h57m25.250898681s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.900823571 (103h57m25.497594593s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.063905047 (103h57m25.660676071s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.073501018 (103h57m25.670272043s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.113442671 (103h57m25.710213697s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.120035234 (103h57m25.716806258s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.191954998 (103h57m25.788726025s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:58:27.450793500 (103h58m25.047564535s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.450883537 (103h58m25.04765457s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: rpc: dial tcp 77.90.31.21:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451074703 (103h58m25.047845735s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: rpc: dial tcp 77.90.31.12:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451150309 (103h58m25.047921346s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: rpc: dial tcp 77.90.31.14:4672: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451331738 (103h58m25.048102771s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:58:27.451342106 (103h58m25.048113141s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: write tcp 159.223.143.39:37834->77.90.31.16:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451368018 (103h58m25.048139029s) [seg_pos={0 0}, upload=116365]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:58:27.484713412 (103h58m25.08148443s) [upload=116365]: Done uploading segments... Mar 10 18:58:27.484792918 (103h58m25.081563946s) [upload=116365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.415806589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X...X...X.............] duration: 1.307862816s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:58:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.307891067s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration: 19.261060794s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:58:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 19.261089476s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X.......X.............X....X] duration: 1m0.036238614s trace: https://jaeger.dev.storj.io/trace/5551066fb24b6624 FAIL (2026-03-10 18:58:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=116365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:57:27.484826534 (103h57m25.081597557s) [upload=116365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:57:27.521721468 (103h57m25.11849248s) [seg_pos={0 0}, upload=116365]: Begin upload segment... Mar 10 18:57:27.521868020 (103h57m25.118639046s) [seg_pos={0 0}, upload=116365]: Done begin upload segment. Mar 10 18:57:27.521902956 (103h57m25.118673979s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.521983909 (103h57m25.118754933s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.523613945 (103h57m25.120384968s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524285299 (103h57m25.121056331s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.524949601 (103h57m25.121720624s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525484827 (103h57m25.12225585s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.525969502 (103h57m25.122740528s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.526481271 (103h57m25.123252294s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527363350 (103h57m25.124134383s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.527903397 (103h57m25.12467443s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.528459198 (103h57m25.125230242s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529239126 (103h57m25.12601015s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.529855918 (103h57m25.126626944s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.530483568 (103h57m25.1272546s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531044521 (103h57m25.127815545s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.531608342 (103h57m25.128379366s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532233632 (103h57m25.12900466s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.532810194 (103h57m25.129581217s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Uploading piece... Mar 10 18:57:27.598647582 (103h57m25.195418592s) [upload=116365]: Next returned nil segment Mar 10 18:57:27.598660671 (103h57m25.195431693s) [upload=116365]: Waiting for error group managing segments... Mar 10 18:57:27.617260011 (103h57m25.214031035s) [noise=true , address=145.79.227.45:4254 , piece_id=UZTZ6H7VQNI7MMVFL5FBST5SLNZQ47XED2RNRB6LKU4LJC7BXWVQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623564891 (103h57m25.22033592s) [noise=true , address=45.91.116.10:4099 , piece_id=NBH2YPXPI3GOPWD32DEUKZPDZ5PENXITZT35UOMX3RLKRWBLJ6TQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.623817360 (103h57m25.220588374s) [noise=true , address=45.91.116.4:4943 , piece_id=QKAVY6T6E5BHQRKTKILCXZUA6LQIVFAZ7DMH7JFH6EEB73SFGN6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.624221744 (103h57m25.220992769s) [noise=true , address=45.91.116.9:4291 , piece_id=3YK352DQH7PT2HCSMBHPIGBQIFXWIXNDE5FX7H2TNZ5334O5D6HQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.627462959 (103h57m25.224233983s) [noise=true , address=145.79.227.33:4890 , piece_id=OW4CDCUND735Q3O55EF4PX4M4TKSIOEWI67PHLISO6QIK6OPHJSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.654127655 (103h57m25.250898681s) [noise=true , address=79.127.219.45:4873 , piece_id=M7FKYFT2QI5ZLRXAZGWJGRFFGIQR6BIXLA4SEZ2D6CF63Z5IYWUQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:27.900823571 (103h57m25.497594593s) [noise=true , address=167.17.72.70:4777 , piece_id=ZPCWKHJR7NYCNNIGCJSLAVXV644MEM5HQY7CWNXWISQJVKLWYNQA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.063905047 (103h57m25.660676071s) [noise=true , address=79.127.205.235:4279 , piece_id=PBNOSAQFGLCKAXXMYUXB4WJADRAVQOHLA53LJO3UHPUA6D3JXU2Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.073501018 (103h57m25.670272043s) [noise=true , address=79.127.205.234:4179 , piece_id=IN4T2HFA4F27KSXVUHO46XZP6K2RHG3YHBBRGTEOCSSQO4DRSWWQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.113442671 (103h57m25.710213697s) [noise=true , address=193.149.164.69:4099 , piece_id=QZFRYTG5ECLVDGGXROBGNXW5SWDERMEOG6YVX62CPMTR3U735O6Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.120035234 (103h57m25.716806258s) [noise=true , address=64.49.13.140:4511 , piece_id=N3JWQXVVCAWUKCYWZ54MA4FM34IORGL436WI37JWPOF7VOMDGM6A, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:57:28.191954998 (103h57m25.788726025s) [noise=true , address=64.49.13.27:4967 , piece_id=PSJ3MO3TEICL6QCFKPXGOQDLGTP7C6YJPQZNCSKJKCIN32YZPZOA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: <nil> Mar 10 18:58:27.450793500 (103h58m25.047564535s) [noise=true , address=77.90.31.18:4815 , piece_id=DZW2JJC3WP52JUX7PQN2GRLNGAGX5ABNYNHLZOR7C2P23CBA3B4Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.450883537 (103h58m25.04765457s) [noise=true , address=77.90.31.21:4900 , piece_id=J3FUPH6SIWIPZUBNEKGJFLP6LY6ICQK6RJJYLMXH2DCNQCTHZWSQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: rpc: dial tcp 77.90.31.21:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451074703 (103h58m25.047845735s) [noise=true , address=77.90.31.12:4593 , piece_id=M64FQFLGQBYYR4SSZT2OHQIQJ342GOLLPFNIZCPKVPMWMWR2I3DA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: rpc: dial tcp 77.90.31.12:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451150309 (103h58m25.047921346s) [noise=true , address=77.90.31.14:4672 , piece_id=UVBCM32Q2OEPKIS2UITNSBTIV4LXYHFN4KNYXTT6IQJFNPLT4QAQ, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: rpc: dial tcp 77.90.31.14:4672: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451331738 (103h58m25.048102771s) [noise=true , address=77.90.31.17:4789 , piece_id=4JYWDROUCOW7ABFSN524QYE5V5TU6TIUOJP6FFRSPPPVZLHTHH3Q, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: context deadline exceeded; context deadline exceeded Mar 10 18:58:27.451342106 (103h58m25.048113141s) [noise=true , address=77.90.31.16:4738 , piece_id=LOIFPP7RKUQSJLSCD54RKTQCRLW7GW2BRY2IPZYNF7WFXI6VIZUA, seg_pos={0 0}, upload=116365]: Done uploading piece. err: ecclient: upload failed (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd, address:77.90.31.16:4738): protocol: write tcp 159.223.143.39:37834->77.90.31.16:4738: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:58:27.451368018 (103h58m25.048139029s) [seg_pos={0 0}, upload=116365]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:58:27.484713412 (103h58m25.08148443s) [upload=116365]: Done uploading segments... Mar 10 18:58:27.484792918 (103h58m25.081563946s) [upload=116365]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.03629341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 930.960293ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:58:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 930.988961ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X..................X..................] duration: 645.185155ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:58:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 645.209115ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X...X...X............] duration: 1.261974057s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:57:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.262016615s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration: 5.407667925s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:57:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.407717948s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 540.695891ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:57:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 540.81365ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration: 4.251333208s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:57:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 4.251367655s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 541.306704ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:57:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 541.348633ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..................X.................] duration: 723.193585ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:57:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 723.230035ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X.............X....] duration: 1.343301892s trace: https://jaeger.dev.storj.io/trace/3a59cfb2d6014c5d FAIL (2026-03-10 18:56:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Mar 10 18:54:00.051608928 (103h53m57.648379956s) [upload=116297]: Got next segment. Inline: false Mar 10 18:54:00.051657088 (103h53m57.648428118s) [upload=116297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:54:00.051665172 (103h53m57.648436204s) [upload=116297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:54:00.094363612 (103h53m57.691134626s) [seg_pos={0 0}, upload=116297]: Begin upload segment... Mar 10 18:54:00.094444424 (103h53m57.691215448s) [seg_pos={0 0}, upload=116297]: Done begin upload segment. Mar 10 18:54:00.094496377 (103h53m57.69126741s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.095179625 (103h53m57.691950659s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096117377 (103h53m57.692888395s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096404698 (103h53m57.693175727s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096884079 (103h53m57.69365511s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097393667 (103h53m57.694164701s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097536619 (103h53m57.694307652s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098010138 (103h53m57.694781171s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098174926 (103h53m57.69494596s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098557982 (103h53m57.695329015s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098797942 (103h53m57.695568965s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099083937 (103h53m57.695855068s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099510235 (103h53m57.696281268s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099634276 (103h53m57.696405308s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.100279270 (103h53m57.697050314s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.103837919 (103h53m57.700608953s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104440865 (103h53m57.701211896s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104899904 (103h53m57.701670933s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.178141749 (103h53m57.77491276s) [upload=116297]: Next returned nil segment Mar 10 18:54:00.178168820 (103h53m57.774939847s) [upload=116297]: Waiting for error group managing segments... Mar 10 18:54:00.189930603 (103h53m57.78670163s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190189723 (103h53m57.786960734s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190412322 (103h53m57.787183334s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.198747756 (103h53m57.795518771s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.202642661 (103h53m57.799413688s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.209451504 (103h53m57.806222516s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.486039656 (103h53m58.082810689s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.510671390 (103h53m58.107442403s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.517154305 (103h53m58.113925316s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.676024477 (103h53m58.272795503s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.679475742 (103h53m58.276246769s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.693360650 (103h53m58.290131664s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.727272577 (103h53m58.324043603s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:55:00.006662901 (103h54m57.603433936s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.006842038 (103h54m57.603613072s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007208693 (103h54m57.603979744s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007375483 (103h54m57.604146519s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008357485 (103h54m57.605128542s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 159.223.143.39:57742->77.90.31.13:4642: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008433591 (103h54m57.605204605s) [seg_pos={0 0}, upload=116297]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:55:00.059752407 (103h54m57.656523439s) [upload=116297]: Done uploading segments... Mar 10 18:55:00.059793948 (103h54m57.656564979s) [upload=116297]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.343330087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X...X...X...........] duration: 1.323432121s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:56:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.323463432s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 538.360008ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:56:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.431776ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X..................X................] duration: 630.540116ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 630.644629ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [............X..X............................................] duration: 5.809707191s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:55:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.809734322s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 527.174957ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:55:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 527.241801ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..................X...............] duration: 934.902629ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:55:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 934.932299ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.......X.............X...] duration: 1.206435142s trace: https://jaeger.dev.storj.io/trace/3a59cfb2d6014c5d FAIL (2026-03-10 18:55:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Mar 10 18:54:00.051608928 (103h53m57.648379956s) [upload=116297]: Got next segment. Inline: false Mar 10 18:54:00.051657088 (103h53m57.648428118s) [upload=116297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:54:00.051665172 (103h53m57.648436204s) [upload=116297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:54:00.094363612 (103h53m57.691134626s) [seg_pos={0 0}, upload=116297]: Begin upload segment... Mar 10 18:54:00.094444424 (103h53m57.691215448s) [seg_pos={0 0}, upload=116297]: Done begin upload segment. Mar 10 18:54:00.094496377 (103h53m57.69126741s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.095179625 (103h53m57.691950659s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096117377 (103h53m57.692888395s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096404698 (103h53m57.693175727s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096884079 (103h53m57.69365511s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097393667 (103h53m57.694164701s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097536619 (103h53m57.694307652s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098010138 (103h53m57.694781171s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098174926 (103h53m57.69494596s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098557982 (103h53m57.695329015s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098797942 (103h53m57.695568965s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099083937 (103h53m57.695855068s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099510235 (103h53m57.696281268s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099634276 (103h53m57.696405308s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.100279270 (103h53m57.697050314s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.103837919 (103h53m57.700608953s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104440865 (103h53m57.701211896s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104899904 (103h53m57.701670933s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.178141749 (103h53m57.77491276s) [upload=116297]: Next returned nil segment Mar 10 18:54:00.178168820 (103h53m57.774939847s) [upload=116297]: Waiting for error group managing segments... Mar 10 18:54:00.189930603 (103h53m57.78670163s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190189723 (103h53m57.786960734s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190412322 (103h53m57.787183334s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.198747756 (103h53m57.795518771s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.202642661 (103h53m57.799413688s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.209451504 (103h53m57.806222516s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.486039656 (103h53m58.082810689s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.510671390 (103h53m58.107442403s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.517154305 (103h53m58.113925316s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.676024477 (103h53m58.272795503s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.679475742 (103h53m58.276246769s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.693360650 (103h53m58.290131664s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.727272577 (103h53m58.324043603s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:55:00.006662901 (103h54m57.603433936s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.006842038 (103h54m57.603613072s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007208693 (103h54m57.603979744s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007375483 (103h54m57.604146519s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008357485 (103h54m57.605128542s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 159.223.143.39:57742->77.90.31.13:4642: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008433591 (103h54m57.605204605s) [seg_pos={0 0}, upload=116297]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:55:00.059752407 (103h54m57.656523439s) [upload=116297]: Done uploading segments... Mar 10 18:55:00.059793948 (103h54m57.656564979s) [upload=116297]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.206459067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X...X...X..........] duration: 1.193588606s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:55:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.193621265s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.............X..X...........................................] duration: 9.738197098s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:55:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 9.738284366s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.......X.............X..] duration: 1.27542208s trace: https://jaeger.dev.storj.io/trace/3a59cfb2d6014c5d FAIL (2026-03-10 18:55:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Mar 10 18:54:00.051608928 (103h53m57.648379956s) [upload=116297]: Got next segment. Inline: false Mar 10 18:54:00.051657088 (103h53m57.648428118s) [upload=116297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:54:00.051665172 (103h53m57.648436204s) [upload=116297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:54:00.094363612 (103h53m57.691134626s) [seg_pos={0 0}, upload=116297]: Begin upload segment... Mar 10 18:54:00.094444424 (103h53m57.691215448s) [seg_pos={0 0}, upload=116297]: Done begin upload segment. Mar 10 18:54:00.094496377 (103h53m57.69126741s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.095179625 (103h53m57.691950659s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096117377 (103h53m57.692888395s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096404698 (103h53m57.693175727s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096884079 (103h53m57.69365511s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097393667 (103h53m57.694164701s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097536619 (103h53m57.694307652s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098010138 (103h53m57.694781171s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098174926 (103h53m57.69494596s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098557982 (103h53m57.695329015s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098797942 (103h53m57.695568965s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099083937 (103h53m57.695855068s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099510235 (103h53m57.696281268s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099634276 (103h53m57.696405308s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.100279270 (103h53m57.697050314s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.103837919 (103h53m57.700608953s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104440865 (103h53m57.701211896s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104899904 (103h53m57.701670933s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.178141749 (103h53m57.77491276s) [upload=116297]: Next returned nil segment Mar 10 18:54:00.178168820 (103h53m57.774939847s) [upload=116297]: Waiting for error group managing segments... Mar 10 18:54:00.189930603 (103h53m57.78670163s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190189723 (103h53m57.786960734s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190412322 (103h53m57.787183334s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.198747756 (103h53m57.795518771s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.202642661 (103h53m57.799413688s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.209451504 (103h53m57.806222516s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.486039656 (103h53m58.082810689s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.510671390 (103h53m58.107442403s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.517154305 (103h53m58.113925316s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.676024477 (103h53m58.272795503s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.679475742 (103h53m58.276246769s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.693360650 (103h53m58.290131664s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.727272577 (103h53m58.324043603s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:55:00.006662901 (103h54m57.603433936s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.006842038 (103h54m57.603613072s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007208693 (103h54m57.603979744s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007375483 (103h54m57.604146519s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008357485 (103h54m57.605128542s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 159.223.143.39:57742->77.90.31.13:4642: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008433591 (103h54m57.605204605s) [seg_pos={0 0}, upload=116297]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:55:00.059752407 (103h54m57.656523439s) [upload=116297]: Done uploading segments... Mar 10 18:55:00.059793948 (103h54m57.656564979s) [upload=116297]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.275447063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration: 2.558625914s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:55:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.558651203s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.......X.............X.] duration: 1.398238135s trace: https://jaeger.dev.storj.io/trace/3a59cfb2d6014c5d FAIL (2026-03-10 18:55:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Mar 10 18:54:00.051608928 (103h53m57.648379956s) [upload=116297]: Got next segment. Inline: false Mar 10 18:54:00.051657088 (103h53m57.648428118s) [upload=116297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:54:00.051665172 (103h53m57.648436204s) [upload=116297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:54:00.094363612 (103h53m57.691134626s) [seg_pos={0 0}, upload=116297]: Begin upload segment... Mar 10 18:54:00.094444424 (103h53m57.691215448s) [seg_pos={0 0}, upload=116297]: Done begin upload segment. Mar 10 18:54:00.094496377 (103h53m57.69126741s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.095179625 (103h53m57.691950659s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096117377 (103h53m57.692888395s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096404698 (103h53m57.693175727s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096884079 (103h53m57.69365511s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097393667 (103h53m57.694164701s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097536619 (103h53m57.694307652s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098010138 (103h53m57.694781171s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098174926 (103h53m57.69494596s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098557982 (103h53m57.695329015s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098797942 (103h53m57.695568965s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099083937 (103h53m57.695855068s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099510235 (103h53m57.696281268s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099634276 (103h53m57.696405308s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.100279270 (103h53m57.697050314s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.103837919 (103h53m57.700608953s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104440865 (103h53m57.701211896s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104899904 (103h53m57.701670933s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.178141749 (103h53m57.77491276s) [upload=116297]: Next returned nil segment Mar 10 18:54:00.178168820 (103h53m57.774939847s) [upload=116297]: Waiting for error group managing segments... Mar 10 18:54:00.189930603 (103h53m57.78670163s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190189723 (103h53m57.786960734s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190412322 (103h53m57.787183334s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.198747756 (103h53m57.795518771s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.202642661 (103h53m57.799413688s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.209451504 (103h53m57.806222516s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.486039656 (103h53m58.082810689s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.510671390 (103h53m58.107442403s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.517154305 (103h53m58.113925316s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.676024477 (103h53m58.272795503s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.679475742 (103h53m58.276246769s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.693360650 (103h53m58.290131664s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.727272577 (103h53m58.324043603s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:55:00.006662901 (103h54m57.603433936s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.006842038 (103h54m57.603613072s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007208693 (103h54m57.603979744s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007375483 (103h54m57.604146519s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008357485 (103h54m57.605128542s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 159.223.143.39:57742->77.90.31.13:4642: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008433591 (103h54m57.605204605s) [seg_pos={0 0}, upload=116297]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:55:00.059752407 (103h54m57.656523439s) [upload=116297]: Done uploading segments... Mar 10 18:55:00.059793948 (103h54m57.656564979s) [upload=116297]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.398278858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X...X...X.........] duration: 1.33183015s trace: https://jaeger.dev.storj.io/trace/3e3462f802bd0715 FAIL (2026-03-10 18:55:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:47:13.131133705 (103h47m10.727904738s) [upload=116175]: Starting upload Mar 10 18:47:13.131204716 (103h47m10.727975809s) [upload=116175]: Done starting upload Mar 10 18:47:13.131321599 (103h47m10.728092622s) [upload=116175]: Uploading segments... Mar 10 18:47:13.131498028 (103h47m10.728269042s) [upload=116175]: Got next segment. Inline: false Mar 10 18:47:13.131549439 (103h47m10.728320462s) [upload=116175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:47:13.131554753 (103h47m10.728325776s) [upload=116175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:47:13.160082729 (103h47m10.756853751s) [seg_pos={0 0}, upload=116175]: Begin upload segment... Mar 10 18:47:13.160171054 (103h47m10.756942077s) [seg_pos={0 0}, upload=116175]: Done begin upload segment. Mar 10 18:47:13.160224126 (103h47m10.756995155s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.160287959 (103h47m10.757058982s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162019643 (103h47m10.758790678s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.162608353 (103h47m10.759379382s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163142708 (103h47m10.759913732s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163503887 (103h47m10.76027492s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.163575954 (103h47m10.760346993s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164036842 (103h47m10.760807865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164118967 (103h47m10.760889993s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164530812 (103h47m10.761301839s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.164752563 (103h47m10.7615236s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165002038 (103h47m10.761773061s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165326057 (103h47m10.762097084s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165449546 (103h47m10.762220572s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165846479 (103h47m10.762617503s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.165877733 (103h47m10.762648758s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166348308 (103h47m10.763119334s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.166840248 (103h47m10.763611272s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Uploading piece... Mar 10 18:47:13.235318918 (103h47m10.832089941s) [upload=116175]: Next returned nil segment Mar 10 18:47:13.235332456 (103h47m10.832103479s) [upload=116175]: Waiting for error group managing segments... Mar 10 18:47:13.241482194 (103h47m10.838253208s) [noise=true , address=145.79.227.11:4746 , piece_id=IPQHMFB7TI6QBYRKBD5QSILG5MSVISYIOHI46Q4XFPIVGXEMFMYQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.241608016 (103h47m10.838379046s) [noise=true , address=79.127.219.45:4865 , piece_id=DNT2XJ27EN7OYU3OSGUVLBGV2XLOC3WGEFRFE24YM7RUVWB3THDA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242020069 (103h47m10.838791092s) [noise=true , address=145.79.227.5:4233 , piece_id=QF5NXVRZRMEIYCVOLABF4K5RLAF6AQDYPKGGZEXXQ2XKD7HAE6KQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.242080152 (103h47m10.838851175s) [noise=true , address=79.127.219.38:4834 , piece_id=HRCTROMY5HCARHYNQHL6UFW7D667ZAFPHH5HKQWIEU4F26VIEE3Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.256022458 (103h47m10.852793481s) [noise=true , address=45.91.116.14:4206 , piece_id=PAGENFX3BMPNE2RQJ7XVCYOCN46Z7K6K6A637XVXWMCLBGSTSVNQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.301466089 (103h47m10.898237119s) [noise=true , address=145.79.227.32:4864 , piece_id=YC4R53EAKWE2HEA2X6KREQHAQQOYF2FL3MFRVX3P6V4EFMTNKNIQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.516165507 (103h47m11.112936531s) [noise=true , address=167.17.72.68:4672 , piece_id=CB2TS4GC6MCT6T3VVVUYDBRAFJDNER6CIDMQI3BQKNBL24K6SMUA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.539446055 (103h47m11.136217082s) [noise=true , address=167.17.72.71:4790 , piece_id=NDFWQKJZ5E7IIQBD4QAHIFLQSMB2NDOHNIU5WLZSVGZSHX3VDWMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.738358444 (103h47m11.335129472s) [noise=true , address=193.149.164.67:4012 , piece_id=W65G5XNHKFPGIDZHC3X77YIK5O4Q4JTDCVHBAFGWADUJB6JUKLCA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.752473842 (103h47m11.349244865s) [noise=true , address=193.149.164.69:4104 , piece_id=OL5YHMN4A5Y4EIVSSKQO25RFRT34D5HSQUMYKVUWJF6DMYCVX75A, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.763441174 (103h47m11.360212197s) [noise=true , address=193.149.164.68:4042 , piece_id=34FVVBDNY3OLD6LJ4PRUQJVK6L6DUMX4BROI7UNTSBYX7CHDOD6Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.764387111 (103h47m11.36115814s) [noise=true , address=193.149.164.71:4318 , piece_id=7WDE5OROP62BN2PYRJSW5QACWOZGHS6H3JSWV4YGK3UMISRIRAMA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:47:13.771769055 (103h47m11.368540082s) [noise=true , address=193.149.164.66:4983 , piece_id=VRXG4YE6RIAZTGIFVRC6EPDUC4I5MTVM6BL35QWGNNZUZ4FBE7MA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: <nil> Mar 10 18:48:13.100100612 (103h48m10.69687164s) [noise=true , address=77.90.31.16:4754 , piece_id=YCVLN5EESISHEV7UN5UQHUNMLCY2A5WETSXZY3XNH7KFO4UNNODA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100324844 (103h48m10.697095871s) [noise=true , address=77.90.31.17:4776 , piece_id=L46MHS6SXUAWGVHTAORD6XEUZK2HNAADS2EFKVO6KZD52VUFVQUQ, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ, address:77.90.31.17:4776): protocol: write tcp 159.223.143.39:39454->77.90.31.17:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100416404 (103h48m10.697187432s) [noise=true , address=77.90.31.12:4603 , piece_id=7EZO6VVXZYB2AFXLDNAGZBOE3Q2PQX57TJBUVXXQ64ZMSQ6T2JQA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100713166 (103h48m10.697484193s) [noise=true , address=77.90.31.11:4561 , piece_id=GKD3QOLFWMUQJ2BWSEZCRIP3SIWK5CGR2OCSD7UG5L4ZMPDQW72Q, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 159.223.143.39:35880->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100860410 (103h48m10.697631433s) [noise=true , address=77.90.31.15:4702 , piece_id=KIPCIMVO4G6CCAOBC5RDGOHNWF2JEJFTLGPDSDDOJ6RQURTZZNEA, seg_pos={0 0}, upload=116175]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: write tcp 159.223.143.39:33634->77.90.31.15:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:48:13.100893545 (103h48m10.697664561s) [seg_pos={0 0}, upload=116175]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:48:13.129150766 (103h48m10.725921853s) [upload=116175]: Done uploading segments... Mar 10 18:48:13.129195755 (103h48m10.725966784s) [upload=116175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.331858102s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.......X.............X] duration: 1m0.055952337s trace: https://jaeger.dev.storj.io/trace/3a59cfb2d6014c5d FAIL (2026-03-10 18:55:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Mar 10 18:54:00.051608928 (103h53m57.648379956s) [upload=116297]: Got next segment. Inline: false Mar 10 18:54:00.051657088 (103h53m57.648428118s) [upload=116297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:54:00.051665172 (103h53m57.648436204s) [upload=116297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:54:00.094363612 (103h53m57.691134626s) [seg_pos={0 0}, upload=116297]: Begin upload segment... Mar 10 18:54:00.094444424 (103h53m57.691215448s) [seg_pos={0 0}, upload=116297]: Done begin upload segment. Mar 10 18:54:00.094496377 (103h53m57.69126741s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.095179625 (103h53m57.691950659s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096117377 (103h53m57.692888395s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096404698 (103h53m57.693175727s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.096884079 (103h53m57.69365511s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097393667 (103h53m57.694164701s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.097536619 (103h53m57.694307652s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098010138 (103h53m57.694781171s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098174926 (103h53m57.69494596s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098557982 (103h53m57.695329015s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.098797942 (103h53m57.695568965s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099083937 (103h53m57.695855068s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099510235 (103h53m57.696281268s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.099634276 (103h53m57.696405308s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.100279270 (103h53m57.697050314s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.103837919 (103h53m57.700608953s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104440865 (103h53m57.701211896s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.104899904 (103h53m57.701670933s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Uploading piece... Mar 10 18:54:00.178141749 (103h53m57.77491276s) [upload=116297]: Next returned nil segment Mar 10 18:54:00.178168820 (103h53m57.774939847s) [upload=116297]: Waiting for error group managing segments... Mar 10 18:54:00.189930603 (103h53m57.78670163s) [noise=true , address=45.91.116.10:4106 , piece_id=UEVJXTAAZCRCK5EFQ57QSWIVFFDWMQAPRGUZBL4EE2SXAQIMA3AQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190189723 (103h53m57.786960734s) [noise=true , address=45.91.116.6:4983 , piece_id=RQMPIZBRTS44HSLWPE6LLBNHGKHAYBF74DCYKDGFBH766HA36QOQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.190412322 (103h53m57.787183334s) [noise=true , address=45.91.116.7:4014 , piece_id=BIRMMCT6W3RIA4OM4I5DM575ZMGXIC2OPEOR6ZZNGBKTWXHPFDFA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.198747756 (103h53m57.795518771s) [noise=true , address=45.91.116.13:4177 , piece_id=NY73TLGGA2SVO4FWX5KUNFLH5MTAZ4CLAEVRRZQYCVYF374GL7NQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.202642661 (103h53m57.799413688s) [noise=true , address=79.127.219.41:4937 , piece_id=B7TCBL3IEUP2EBN3IG2VWOIBMDLTJX6CM533AZ3QVJQUMSI63EWQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.209451504 (103h53m57.806222516s) [noise=true , address=145.79.227.15:4901 , piece_id=ZSGPQE5JR2LLNHXFNXJXN2U3GMWLKTIKTAE3W6GKCBSWE4XQDHSA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.486039656 (103h53m58.082810689s) [noise=true , address=167.17.72.71:4794 , piece_id=BTUZO3O4IEXXK3NOFM7TVHYXREDLYRLYZNRDTN7GBK5ED25NPUQQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.510671390 (103h53m58.107442403s) [noise=true , address=167.17.72.68:4665 , piece_id=WKZ5BOGPSSZ43PC2EXHBMZY3VHTGNBBMNVPOP3CW3S2DFYZQOLCQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.517154305 (103h53m58.113925316s) [noise=true , address=167.17.72.70:4772 , piece_id=KLZ6C5LZ2LT7GWAJXFAIKTQHRYMNA327KNJVGAC3B7ZS47EFOHIQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.676024477 (103h53m58.272795503s) [noise=true , address=193.149.164.71:4317 , piece_id=VL2S4KZNJVCQ556EPKU4KUOQECI2NEPCOMNOPTF5OQOZHGDG4EUA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.679475742 (103h53m58.276246769s) [noise=true , address=79.127.205.235:4276 , piece_id=CSPEWIKYXSJLE63VK63HQPADXU55IWOORQJHJRL23X73F6FN4HFQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.693360650 (103h53m58.290131664s) [noise=true , address=193.149.164.69:4104 , piece_id=CVDZ5JXX6VPSRH7PQRA5NFNRHUQKAIY5OHPPHY6U5IBURI26EMKA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:54:00.727272577 (103h53m58.324043603s) [noise=true , address=193.149.164.68:4061 , piece_id=JMOAJPUZFNADZTLSDSKC6LSEU2SUEC5OHBRMFS4X537FMRDE2ESQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: <nil> Mar 10 18:55:00.006662901 (103h54m57.603433936s) [noise=true , address=77.90.31.17:4795 , piece_id=VN6HMNLAFXXH42WRSRGKZDARNA46L2G6PABDV5DCU5URUEUKQM3Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: rpc: dial tcp 77.90.31.17:4795: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.006842038 (103h54m57.603613072s) [noise=true , address=77.90.31.14:4690 , piece_id=EGTK2USUBVETC75NA7SINXZTZMO44J4B72U3T7IQO2SOLKMSLBAQ, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007208693 (103h54m57.603979744s) [noise=true , address=77.90.31.19:4871 , piece_id=CBW4I5CMMVB2AE3KKX67LNFWDA5MODSPUDUZCSTAORNGRZOIKH4Q, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.007375483 (103h54m57.604146519s) [noise=true , address=77.90.31.10:4536 , piece_id=24AYLRJ5ELKAR43JLEPACUW3E6AZBHWGCI434TGJG26A2BPUZBZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008357485 (103h54m57.605128542s) [noise=true , address=77.90.31.13:4642 , piece_id=5SIY62MHJ33RGF7TTSJA4OMOJ3M4DLZXDXAOQDMRVT3NMTTEMHZA, seg_pos={0 0}, upload=116297]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 159.223.143.39:57742->77.90.31.13:4642: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:55:00.008433591 (103h54m57.605204605s) [seg_pos={0 0}, upload=116297]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:55:00.059752407 (103h54m57.656523439s) [upload=116297]: Done uploading segments... Mar 10 18:55:00.059793948 (103h54m57.656564979s) [upload=116297]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.055982322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 661.227401ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 18:54:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 661.252687ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration: 6.944763604s trace: https://jaeger.dev.storj.io/trace/62b857fbddd8c134 FAIL (2026-03-10 18:54:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== TJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.678723726 (103h22m15.275494752s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679171303 (103h22m15.275942329s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.679573269 (103h22m15.276344303s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680028932 (103h22m15.27679996s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680473046 (103h22m15.277244178s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.680982200 (103h22m15.277753227s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681478081 (103h22m15.27824911s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.681895554 (103h22m15.278666585s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682340386 (103h22m15.279111418s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.682758716 (103h22m15.279529749s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:17.683342542 (103h22m15.280113571s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Uploading piece... Mar 10 18:22:18.102504696 (103h22m15.699275711s) [upload=115702]: Next returned nil segment Mar 10 18:22:18.102520582 (103h22m15.699291606s) [upload=115702]: Waiting for error group managing segments... Mar 10 18:22:18.286488354 (103h22m15.883259367s) [noise=true , address=185.167.204.206:4037 , piece_id=TEYUSWWSWXEKS2TUKAXH7NXD57TQKEZLPS5IOYQVC54ZTY2LEHGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.291859985 (103h22m15.888631012s) [noise=true , address=81.171.14.70:4031 , piece_id=43ZJLSQWG6ML6VS75ZQ7DFAI56Z7G52P3GXMFKHDQT4ZJKJ333QA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.297964517 (103h22m15.894735561s) [noise=true , address=89.149.201.147:4005 , piece_id=CFTY5WE3SZLKOPH6KT4ZD3M2A7ZCIOH7BCUUPGTYM6NRFNJC2BZA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.307699767 (103h22m15.904470781s) [noise=true , address=89.149.201.147:4010 , piece_id=43BOZGJPSID77Q2EY7ULJQIK45WZ4IEZZ4HTFQ43J7TP2SIXJQKQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.312994778 (103h22m15.909765791s) [noise=true , address=185.167.204.205:4017 , piece_id=GLOJ3UR3HQZRVVZ3BMUAXSQBK6YJ4MX5EBGPH6LNET7JDH4RWEXQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.319374438 (103h22m15.916145452s) [noise=true , address=81.171.14.70:4048 , piece_id=JYSM26GYUVQK4QTC444MOC4KXPWBU5OUVH5QPWQGP72AGFBC2MIQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.341507663 (103h22m15.938278696s) [noise=true , address=185.167.204.206:4054 , piece_id=GCJ4ZXR6YMUGI5BG2XOYP4FKTTTSPGVZ5UYS4BOMALGHCSBFQO7Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.351324302 (103h22m15.948095328s) [noise=true , address=81.171.14.70:4032 , piece_id=GPOUM37IH7EL7YANALUZPAQOQ2MHAFOZSVWM7VLDPCNRODOKOO5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.354375553 (103h22m15.951146587s) [noise=true , address=185.167.204.205:4032 , piece_id=YA3TR364FWAMYKQUROV4PYQWTGIXM7SLAUM3BN2FRVBW3MVILHDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.355210000 (103h22m15.951981024s) [noise=true , address=89.149.201.147:4020 , piece_id=AGOZMT6EYQKSFRYOJW4QAV2NMSPM4Q5LAI2SA276EKYK4X5ZGP5Q, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.364878440 (103h22m15.961649464s) [noise=true , address=185.167.204.206:4039 , piece_id=4LSO5WGH5LQ5NIDHOHOACKVAYY37JEZHAUX66MEJ4FIGZ5LWDRFA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.366270355 (103h22m15.963041377s) [noise=true , address=185.167.204.206:4049 , piece_id=JPGUJCUADQNZU5KJAHY4FXUTKJVSEEGMZQ7HNUEWOIWYBJ3LYPEQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.383218910 (103h22m15.979989919s) [noise=true , address=185.167.204.205:4025 , piece_id=2BA453RCULDYAHVTMMVFTPT5MRC4VJNBKK3IHUOLSOMERCHA6LKA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.385734313 (103h22m15.982505326s) [noise=true , address=185.167.204.206:4062 , piece_id=LOWNTJEFWAHKTTCYX2TMYQOT5XKRZQQAUWT7XHPFBHBHLDJFYQQQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.386456163 (103h22m15.983227177s) [noise=true , address=185.167.204.206:4071 , piece_id=XT7Y2BSAL4MSA4P6GXDWGQ3R7TGBZT4ON7KAJNVUMBHYPPYU4YTA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389060733 (103h22m15.985831747s) [noise=true , address=185.167.204.205:4001 , piece_id=5T7AKGPWFTSCKDMKK3BOLGSIH3OAVNAZB3D5ILXXSYOR2CA2NFHQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389075564 (103h22m15.985846595s) [noise=true , address=185.167.204.206:4068 , piece_id=ILUQHWCZH6TERJONKPNEQCZKFZF7TD757AGIQU3DD7HWSKQ52NDA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.389787069 (103h22m15.986558092s) [noise=true , address=185.167.204.205:4013 , piece_id=ZYGKKFJQRQZZVEO7DQ67E3ILC66TO5EQBXBAFMKRS2CMKKOKLLVA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.392844303 (103h22m15.989615315s) [noise=true , address=185.167.204.205:4030 , piece_id=OYNJ3OVLH2SXV4I432BJOXP3YPCAOML62OTFNYBFMN47PQ5UGWAA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.395040677 (103h22m15.99181169s) [noise=true , address=185.167.204.206:4053 , piece_id=D72PWJW544HPN6C4C6YIIQBUP7ZIWIN4VCTSYRLGUSCIEUHBWMQA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397224921 (103h22m15.993995937s) [noise=true , address=185.167.204.205:4036 , piece_id=WZ5MCIX5JEJ5QNDULLGIVXNKQBVFEX7ATOQNBTETSOW2WCNWYSJQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.397305588 (103h22m15.994076606s) [noise=true , address=89.149.201.147:4016 , piece_id=ELN7QQHMWZICYFEWCRGUR5OUEVLKDNKHK5Y3KEQXDFRCE3SWEB2A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:18.423243063 (103h22m16.020014089s) [noise=true , address=185.167.204.206:4067 , piece_id=XWH6ZTD6S3TN6TIFN5S4ZIBNR44VOYCB7UCKXOMBBM2MPMNSXUGA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:24.985281962 (103h22m22.582052989s) [noise=true , address=185.167.204.205:4007 , piece_id=GDYOPUY5PJFCYLU7OYPBVTW424GT7FP77TMRYKYNKUWXZS6ENAYQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:22:30.233899651 (103h22m27.830670686s) [noise=true , address=185.167.204.205:4020 , piece_id=5GIMRHZGF2ZEFWLBL4U6KF4DKZ53JZ7PNWB4XTEQW5XCESHNZY5A, seg_pos={0 0}, upload=115702]: Done uploading piece. err: <nil> Mar 10 18:23:17.302478912 (103h23m14.899249945s) [noise=true , address=185.167.204.205:4024 , piece_id=MCKCFKQPVK4H7YWVNQBYD7VLK26D4ABODUMWYEESU22S2GZ2X4UA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57608->185.167.204.205:4024: use of closed network connection Mar 10 18:23:17.302510476 (103h23m14.899281492s) [noise=true , address=89.149.201.147:4014 , piece_id=VZUX5U4EDER24ANUL7OGMVS5JWIE45RT5HINRYU662ULMKG3Q6SA, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: write tcp 159.223.143.39:39206->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:23:17.302719428 (103h23m14.89949046s) [noise=true , address=185.167.204.206:4058 , piece_id=LZ6PUPONPGDVBLY3IIUUFUZENZTSTWNMCBEK4GVI3Z2RQXICZSDQ, seg_pos={0 0}, upload=115702]: Done uploading piece. err: ecclient: upload failed (node:12cJjosS5PjPTn1t4TF4Xw7sL6c2kofhJyiLtitrnDy9V2PZFM7, address:185.167.204.206:4058): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 18:23:17.302761551 (103h23m14.899532565s) [seg_pos={0 0}, upload=115702]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 18:23:17.401134721 (103h23m14.997905738s) [upload=115702]: Done uploading segments... Mar 10 18:23:17.401207170 (103h23m14.997978196s) [upload=115702]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.944790172s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..................X..............] duration: 660.981753ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 18:54:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 661.013508ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)