Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X] duration: 30.038491985s FAIL (2025-07-27 00:20:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.038545626s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X.X........X...XX.............................] duration: 38.053913073s FAIL (2025-07-27 00:20:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.053934382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029390596s FAIL (2025-07-27 00:20:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ul 27 00:19:25.706249288 (2h27m41.487182654s) [seg_pos={0 0}, upload=2270]: Done begin upload segment. Jul 27 00:19:25.706293236 (2h27m41.487226619s) [noise=true , address=64.49.13.140:4514 , piece_id=H5XDPXAAXSOADAMLEDVSZM7ZWNAGZ3UUCOJBPPY2HEZJM73O3YRA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.706312245 (2h27m41.487245629s) [noise=true , address=104.36.82.53:4149 , piece_id=4Z3PWW6ORAURBN2QXMP3X2PHISDIOLBEPTX4PYF5HJFIGOSSP6EA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.706775540 (2h27m41.487708927s) [noise=true , address=79.127.205.246:4258 , piece_id=NIB3JHTSBTGJCFIR66BEQ5EQVMG6WOUQOQFYGDLZHBA7RXUNJGRA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.706911155 (2h27m41.487844541s) [noise=true , address=77.90.31.12:4615 , piece_id=THUY33GOVCFFMPLRYB73VSMWMV5XG7PV3GREJ2S2QYYBII2QZHJQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707336964 (2h27m41.488270349s) [noise=true , address=79.127.205.245:4242 , piece_id=Z5YHTFBGQ6N4NBKQ4GGLGXXROTANZIWRPQXJATSRXU6QDRRWRLRA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707404411 (2h27m41.488337797s) [noise=true , address=156.146.43.225:4448 , piece_id=L3UBVGMRBW2LJ4IEKFDIW2BQQSWRTAESUI4VELS4DQH4D4ZC65MQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707758778 (2h27m41.488692166s) [noise=true , address=104.36.82.39:4755 , piece_id=S2CUQL25DE7CPJC3N5YV3O5SOA6SRZI6J4LKJCG4PDJP2TX3HNWA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707854359 (2h27m41.488787749s) [noise=true , address=79.127.219.38:4010 , piece_id=3XWNV2ZLKPEJBWSAFU7MAYKXPE36QQT2Q4TQXUZNGTD6FYR2MLMQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.708162682 (2h27m41.489096076s) [noise=true , address=79.127.205.234:4604 , piece_id=WY3NEWC46GRELOHU6CR2N6HQD4BXFLZVVNREZ452X7PAZFLDMVWA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.708544862 (2h27m41.489478248s) [noise=true , address=79.127.220.98:4422 , piece_id=WKVGGMAFY3WFBJ53ZXDWIAZB3MLBUQHVFU2CTI4R5YKU2ZSSL5SQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.708726535 (2h27m41.489659938s) [noise=true , address=104.36.82.11:4116 , piece_id=PSOPMK33F6S7FTCRVSSP2RFUFPPTH36Z5WWFXGUJQH5YAZYPARAQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.709121937 (2h27m41.490055324s) [noise=true , address=104.36.82.54:4189 , piece_id=IN6RS3SW7UVXELEO3WXD3ZKO7LZBAIYYSQ2HPADY3JMKKKDGJQVQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.711105701 (2h27m41.492039088s) [noise=true , address=77.90.31.19:4875 , piece_id=L6RKXJCWEUCYLD6277BBLIZPNKSFR7KHF6QPWHSDHVKNGX6543HA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.711232622 (2h27m41.492166005s) [noise=true , address=104.36.82.46:4512 , piece_id=CH3PXUGB6V53UYRLDXDYP3TXDG6BZP3NJEN3LHL6X37E5QRS4G3A, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.711871366 (2h27m41.492804755s) [noise=true , address=79.127.205.235:4681 , piece_id=YNMMADATC3QZA5FOYS3S3DYZRYVG7XR6N6NGJUD5WLDMGCIWWWEA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.713814558 (2h27m41.494747946s) [noise=true , address=77.90.31.10:4540 , piece_id=I255I52UZO2YFNHA23ADQK4LFGS3IONE2VVMC3H34T6VTNPEZKQA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.714466206 (2h27m41.495399594s) [noise=true , address=104.36.82.47:4523 , piece_id=ODCOGUJWDQUYE6U7SHNYBBCHW7Z7HPLHAOVLBSNDWTAUML4STD2A, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.715059784 (2h27m41.495993171s) [noise=true , address=79.127.201.215:4334 , piece_id=TH6F2VG27ZCD3H3CNVAX2GJK2XASQWGR5KQLDLFSKCAROG6K6F7Q, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.802847787 (2h27m41.583781173s) [upload=2270]: Next returned nil segment Jul 27 00:19:25.802885643 (2h27m41.583819037s) [upload=2270]: Waiting for error group managing segments... Jul 27 00:19:25.811137945 (2h27m41.592071307s) [noise=true , address=79.127.219.38:4010 , piece_id=3XWNV2ZLKPEJBWSAFU7MAYKXPE36QQT2Q4TQXUZNGTD6FYR2MLMQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:25.957278170 (2h27m41.738211546s) [noise=true , address=79.127.220.98:4422 , piece_id=WKVGGMAFY3WFBJ53ZXDWIAZB3MLBUQHVFU2CTI4R5YKU2ZSSL5SQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE, address:79.127.220.98:4422): protocol: storage node overloaded, request limit: 2000 Jul 27 00:19:26.063714809 (2h27m41.844648185s) [noise=true , address=79.127.201.215:4334 , piece_id=TH6F2VG27ZCD3H3CNVAX2GJK2XASQWGR5KQLDLFSKCAROG6K6F7Q, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.336237481 (2h27m42.11717086s) [noise=true , address=79.127.205.246:4258 , piece_id=NIB3JHTSBTGJCFIR66BEQ5EQVMG6WOUQOQFYGDLZHBA7RXUNJGRA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.347751756 (2h27m42.128685126s) [noise=true , address=79.127.205.234:4604 , piece_id=WY3NEWC46GRELOHU6CR2N6HQD4BXFLZVVNREZ452X7PAZFLDMVWA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.382836995 (2h27m42.163770363s) [noise=true , address=79.127.205.235:4681 , piece_id=YNMMADATC3QZA5FOYS3S3DYZRYVG7XR6N6NGJUD5WLDMGCIWWWEA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.831735028 (2h27m42.612668415s) [noise=true , address=79.127.205.245:4242 , piece_id=Z5YHTFBGQ6N4NBKQ4GGLGXXROTANZIWRPQXJATSRXU6QDRRWRLRA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:27.768486357 (2h27m43.549419723s) [noise=true , address=64.49.13.140:4514 , piece_id=H5XDPXAAXSOADAMLEDVSZM7ZWNAGZ3UUCOJBPPY2HEZJM73O3YRA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:29.992944619 (2h27m45.773877996s) [noise=true , address=156.146.43.225:4448 , piece_id=L3UBVGMRBW2LJ4IEKFDIW2BQQSWRTAESUI4VELS4DQH4D4ZC65MQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:34.937964336 (2h27m50.718897705s) [noise=true , address=77.90.31.12:4615 , piece_id=THUY33GOVCFFMPLRYB73VSMWMV5XG7PV3GREJ2S2QYYBII2QZHJQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:38.553526460 (2h27m54.334459829s) [noise=true , address=77.90.31.19:4875 , piece_id=L6RKXJCWEUCYLD6277BBLIZPNKSFR7KHF6QPWHSDHVKNGX6543HA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:20:25.585109159 (2h28m41.366042536s) [noise=true , address=104.36.82.46:4512 , piece_id=CH3PXUGB6V53UYRLDXDYP3TXDG6BZP3NJEN3LHL6X37E5QRS4G3A, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:25.585469865 (2h28m41.366403241s) [noise=true , address=104.36.82.39:4755 , piece_id=S2CUQL25DE7CPJC3N5YV3O5SOA6SRZI6J4LKJCG4PDJP2TX3HNWA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1cmNqubJ82X6bTLTn5zmMD7HhVDKyP8SH9vo2ZYq5S6GmX6Wpx, address:104.36.82.39:4755): protocol: write tcp 159.223.143.39:39766->104.36.82.39:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585534272 (2h28m41.366467639s) [noise=true , address=104.36.82.11:4116 , piece_id=PSOPMK33F6S7FTCRVSSP2RFUFPPTH36Z5WWFXGUJQH5YAZYPARAQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:123aoo9g11aJP9hiL8shu2dx44tjyKzEWqPUjHUdXtR7LLFyng8, address:104.36.82.11:4116): protocol: write tcp 159.223.143.39:39408->104.36.82.11:4116: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585574078 (2h28m41.366507462s) [noise=true , address=77.90.31.10:4540 , piece_id=I255I52UZO2YFNHA23ADQK4LFGS3IONE2VVMC3H34T6VTNPEZKQA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: write tcp 159.223.143.39:60792->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585615670 (2h28m41.366549032s) [noise=true , address=104.36.82.53:4149 , piece_id=4Z3PWW6ORAURBN2QXMP3X2PHISDIOLBEPTX4PYF5HJFIGOSSP6EA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585631670 (2h28m41.366565051s) [noise=true , address=104.36.82.47:4523 , piece_id=ODCOGUJWDQUYE6U7SHNYBBCHW7Z7HPLHAOVLBSNDWTAUML4STD2A, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 159.223.143.39:40386->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585787287 (2h28m41.366720667s) [noise=true , address=104.36.82.54:4189 , piece_id=IN6RS3SW7UVXELEO3WXD3ZKO7LZBAIYYSQ2HPADY3JMKKKDGJQVQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:25.585835115 (2h28m41.366768492s) [seg_pos={0 0}, upload=2270]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:20:25.611663486 (2h28m41.392596855s) [upload=2270]: Done uploading segments... Jul 27 00:20:25.611688524 (2h28m41.392621893s) [upload=2270]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.029407984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.XXX...............X..X......X.XX....X...X...............] duration: 28.002856191s FAIL (2025-07-27 00:20:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.003134514s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X..............X.X........X...XX............................] duration: 46.693315509s FAIL (2025-07-27 00:19:49 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.693342694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....] duration: 557.523641ms FAIL (2025-07-27 00:19:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 557.55777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.028967626s FAIL (2025-07-27 00:19:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 13628s) [upload=2253]: Uploading segments... Jul 27 00:18:25.505447450 (2h26m41.286380826s) [upload=2253]: Got next segment. Inline: false Jul 27 00:18:25.505497438 (2h26m41.286430814s) [upload=2253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:18:25.505503257 (2h26m41.286436634s) [upload=2253]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:18:25.542470391 (2h26m41.323403766s) [seg_pos={0 0}, upload=2253]: Begin upload segment... Jul 27 00:18:25.542524862 (2h26m41.323458237s) [seg_pos={0 0}, upload=2253]: Done begin upload segment. Jul 27 00:18:25.542551327 (2h26m41.323484708s) [noise=true , address=79.127.205.234:4620 , piece_id=LLVX4NZ2Y72PBSEIHTB4FYEI3NWRVLG36CPAEGVILGQXUFIRZ3GA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.542574512 (2h26m41.323507887s) [noise=true , address=77.90.31.16:4755 , piece_id=UEBN6CJTXQUTGCRNLQTNVU7W6AEQWRWWEJYOM5EB5TQWETRUPYTA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.542978014 (2h26m41.323911399s) [noise=true , address=156.146.43.225:4448 , piece_id=6TCBZVCNR7UWKUYZQ32J7GPLXITM6XAQ2426ZIFLREB4D5RPPQFA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.542999510 (2h26m41.323932888s) [noise=true , address=79.127.201.215:4345 , piece_id=NOUYE6SVY5G7BX2N26TII4IP62B27SB2FAQ6VXMVCAJLJBTCU76A, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543392469 (2h26m41.324325855s) [noise=true , address=77.90.31.15:4717 , piece_id=Q4ATETGUL4XFWGLQNBYT7QBJGWSUCQSLU6NITNVHGQTF6JPVTCSQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543407344 (2h26m41.324340722s) [noise=true , address=79.127.220.101:4041 , piece_id=3XWLWEOYSODRVMXWZ4URMKIFDRCFOTDZB3WZJJIVIRHO42FAAZ2A, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543810392 (2h26m41.324743773s) [noise=true , address=77.90.31.21:4910 , piece_id=A663M76HRVEMVJ6KRUKEOFBOQ7PD44OU5CI7RFX5RUFKDOPCTBWQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543856966 (2h26m41.32479035s) [noise=true , address=79.127.205.233:4650 , piece_id=VSOY4IGX4NDAKQ2ZEWZTGUFU65JIWXSZEV7EGCBKU7F6AVHZF3RA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544407014 (2h26m41.325340398s) [noise=true , address=64.49.13.138:4482 , piece_id=XNQYZQUTUY4FD26EP5DMDRZXH6UEHWEBIEJNT4QORQJK7TOKXSLQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544458776 (2h26m41.325392158s) [noise=true , address=64.49.13.140:4508 , piece_id=HS4TXGBDK32Q45LXM5N2CBKWFH2JWLZXP3KEZWEDKZHVYA6BNNJQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544775396 (2h26m41.325708775s) [noise=true , address=77.90.31.17:4787 , piece_id=PDAC4LUZWBJDHUBYT6JSU2NC7ZK3K3N2T6UBSCPPF2WTVJ2WYODQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544895001 (2h26m41.325828392s) [noise=true , address=104.36.82.47:4524 , piece_id=S3FOVUTRU47GSK7W4JGGD7SE6CPSA27WMXC356T6G4WH3VYFS2DQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.545225065 (2h26m41.32615845s) [noise=true , address=104.36.82.46:4484 , piece_id=RCW4XEBEVFGHJKKOFDFSVV5P4FXESZVBK4V3N5L6GIJKT4HEZTQA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.545264602 (2h26m41.326197988s) [noise=true , address=79.127.205.235:4692 , piece_id=MDJBEPZURM4QBD45YFBDAVGK22FBMHZF5IIN5CQ3S56COQR3WGPQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.546316170 (2h26m41.327249556s) [noise=true , address=79.127.205.246:4283 , piece_id=LVZ4FH2P5XR4INTRI5LGG4CW23NIOYQF4LJ7KBSMNSO2NJ7UIFBQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.546671841 (2h26m41.327605231s) [noise=true , address=104.36.82.53:4141 , piece_id=477OMFJON5AN4TJDGKLKUB25DBPBX2ZCEXNIMTGEOMBYAP7DASUQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.546891803 (2h26m41.327825188s) [noise=true , address=79.127.205.245:4247 , piece_id=KUO6XSRZGHP4YHNB5S35BMC4GIZ5PI3P2LLZULM52EPCCLUM6JEQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.547610759 (2h26m41.328544141s) [noise=true , address=104.36.82.56:4257 , piece_id=ZSLRRQBIZ3R3FPSZTCKSWJ5CQI7FRZMDFKTVBN5W4EA5DEWQVL2Q, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.631350909 (2h26m41.412284293s) [upload=2253]: Next returned nil segment Jul 27 00:18:25.631373306 (2h26m41.412306694s) [upload=2253]: Waiting for error group managing segments... Jul 27 00:18:25.677379328 (2h26m41.458312697s) [noise=true , address=77.90.31.15:4717 , piece_id=Q4ATETGUL4XFWGLQNBYT7QBJGWSUCQSLU6NITNVHGQTF6JPVTCSQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:25.679049752 (2h26m41.459983128s) [noise=true , address=77.90.31.16:4755 , piece_id=UEBN6CJTXQUTGCRNLQTNVU7W6AEQWRWWEJYOM5EB5TQWETRUPYTA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.129934537 (2h26m41.910867903s) [noise=true , address=79.127.205.233:4650 , piece_id=VSOY4IGX4NDAKQ2ZEWZTGUFU65JIWXSZEV7EGCBKU7F6AVHZF3RA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.205940754 (2h26m41.986874118s) [noise=true , address=79.127.205.234:4620 , piece_id=LLVX4NZ2Y72PBSEIHTB4FYEI3NWRVLG36CPAEGVILGQXUFIRZ3GA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.206893674 (2h26m41.987827037s) [noise=true , address=79.127.205.235:4692 , piece_id=MDJBEPZURM4QBD45YFBDAVGK22FBMHZF5IIN5CQ3S56COQR3WGPQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.207737374 (2h26m41.988670752s) [noise=true , address=79.127.205.246:4283 , piece_id=LVZ4FH2P5XR4INTRI5LGG4CW23NIOYQF4LJ7KBSMNSO2NJ7UIFBQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.230425581 (2h26m42.011358942s) [noise=true , address=79.127.205.245:4247 , piece_id=KUO6XSRZGHP4YHNB5S35BMC4GIZ5PI3P2LLZULM52EPCCLUM6JEQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.231838104 (2h26m42.01277148s) [noise=true , address=64.49.13.140:4508 , piece_id=HS4TXGBDK32Q45LXM5N2CBKWFH2JWLZXP3KEZWEDKZHVYA6BNNJQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:28.066848111 (2h26m43.847781489s) [noise=true , address=156.146.43.225:4448 , piece_id=6TCBZVCNR7UWKUYZQ32J7GPLXITM6XAQ2426ZIFLREB4D5RPPQFA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:29.603624553 (2h26m45.384557923s) [noise=true , address=79.127.220.101:4041 , piece_id=3XWLWEOYSODRVMXWZ4URMKIFDRCFOTDZB3WZJJIVIRHO42FAAZ2A, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:31.883139666 (2h26m47.664073043s) [noise=true , address=64.49.13.138:4482 , piece_id=XNQYZQUTUY4FD26EP5DMDRZXH6UEHWEBIEJNT4QORQJK7TOKXSLQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:41.511553208 (2h26m57.292486586s) [noise=true , address=79.127.201.215:4345 , piece_id=NOUYE6SVY5G7BX2N26TII4IP62B27SB2FAQ6VXMVCAJLJBTCU76A, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:19:25.475283152 (2h27m41.256216537s) [noise=true , address=77.90.31.21:4910 , piece_id=A663M76HRVEMVJ6KRUKEOFBOQ7PD44OU5CI7RFX5RUFKDOPCTBWQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: rpc: dial tcp 77.90.31.21:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475417500 (2h27m41.256350878s) [noise=true , address=104.36.82.53:4141 , piece_id=477OMFJON5AN4TJDGKLKUB25DBPBX2ZCEXNIMTGEOMBYAP7DASUQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475533294 (2h27m41.256466674s) [noise=true , address=77.90.31.17:4787 , piece_id=PDAC4LUZWBJDHUBYT6JSU2NC7ZK3K3N2T6UBSCPPF2WTVJ2WYODQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475553828 (2h27m41.256487198s) [noise=true , address=104.36.82.56:4257 , piece_id=ZSLRRQBIZ3R3FPSZTCKSWJ5CQI7FRZMDFKTVBN5W4EA5DEWQVL2Q, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: rpc: dial tcp 104.36.82.56:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475615507 (2h27m41.256548877s) [noise=true , address=104.36.82.46:4484 , piece_id=RCW4XEBEVFGHJKKOFDFSVV5P4FXESZVBK4V3N5L6GIJKT4HEZTQA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475646375 (2h27m41.256579739s) [noise=true , address=104.36.82.47:4524 , piece_id=S3FOVUTRU47GSK7W4JGGD7SE6CPSA27WMXC356T6G4WH3VYFS2DQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: rpc: dial tcp 104.36.82.47:4524: i/o timeout; context deadline exceeded Jul 27 00:19:25.475685142 (2h27m41.256618528s) [seg_pos={0 0}, upload=2253]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:19:25.499876587 (2h27m41.280809969s) [upload=2253]: Done uploading segments... Jul 27 00:19:25.499902437 (2h27m41.280835808s) [upload=2253]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.02898918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XX..............X.X........X...XX...........................] duration: 45.150676992s FAIL (2025-07-27 00:19:02 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.150710876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..] duration: 2.033144609s FAIL (2025-07-27 00:19:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.033299529s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X....] duration: 1.97645334s FAIL (2025-07-27 00:18:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.976479369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.XXX...............X..X......X.XX....X...X..............] duration: 2.046254052s FAIL (2025-07-27 00:18:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.046358918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X...] duration: 2.047593574s FAIL (2025-07-27 00:18:34 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.047648109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030855327s FAIL (2025-07-27 00:18:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 3ZBYYPWQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.433429709 (2h25m41.21436309s) [noise=true , address=104.36.82.53:4121 , piece_id=5NX3GTJG4GVSAD3ELLCHAF2O2AKIZS5OLCYWFI2WXAYYEYXQB4VQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.433808462 (2h25m41.214741853s) [noise=true , address=79.127.219.47:4048 , piece_id=3IRM2Z4VJDDUUUNS55IEJ6QZDL3I7Z2VVXNME7KTK7BJ7KX2ZPAA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.434201983 (2h25m41.215135367s) [noise=true , address=156.146.43.225:4448 , piece_id=FOD34AK2IRA54YXSIQ677YDARBMQZEVXCVYXV4O54VRTLSXEGXQQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.434547297 (2h25m41.21548068s) [noise=true , address=79.127.205.235:4669 , piece_id=OYG3OI5WNHYTGHNXRC5XWAZ7K5HXMMPXHE2FCG5WJEYBESEPHP3Q, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.434944479 (2h25m41.215877905s) [noise=true , address=104.36.82.54:4169 , piece_id=P5K5LAJSGGIG45SCX5P74A7ZH6BBKPRO26TFVF7I523LOWWOCSRQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435089878 (2h25m41.216023258s) [noise=true , address=104.36.82.47:4540 , piece_id=IKVWDCQM7PCG3BK4IFGLEZTHVUTFNOLYGWJCONO7OJGO27KXGWGQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435415377 (2h25m41.216348765s) [noise=true , address=79.127.201.214:4289 , piece_id=3ZTNGA4JFHXXPA4SNPSDTSE4A7BO2XEDYDPDC3FYH3DPGBT5RDYA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435809500 (2h25m41.21674289s) [noise=true , address=79.127.205.246:4280 , piece_id=U2NWGWBUZ56YBIX6JCS4WWAWPJKQL4H6U4OEJLMRGLYMXRPFCDLA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435832958 (2h25m41.216766344s) [noise=true , address=104.36.82.56:4252 , piece_id=4C2CKS62DVOK4FD7ANAYPFSWA44RPIFCPAQ2JMSKGBCQVWBDT4BA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.436290772 (2h25m41.217224135s) [noise=true , address=79.127.220.101:4041 , piece_id=V2HCCLSBVIJ3EHKLGZQTUG2NBKLGXBTLWASNU2YTAUFJLMEJTC2Q, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.436645989 (2h25m41.217579376s) [noise=true , address=104.36.82.46:4486 , piece_id=IRF7MBR3UJ2I66DUAE7WRDYIXTT4KN3B744BR3ZSD5M6KMNX2YGQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.437068321 (2h25m41.218001713s) [noise=true , address=64.49.13.154:4927 , piece_id=4EU77ZGSLMYQL3IW4EITP6HNOKCTV34UE6IXYZQAAGLGGGTSYMHA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.437470784 (2h25m41.218404164s) [noise=true , address=45.91.116.14:4206 , piece_id=SJ242HH354E2I64RMLODFM5ILO4JEIQJ533LESMFDMO43QC6SYZQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.437806428 (2h25m41.218739811s) [noise=true , address=79.127.201.215:4360 , piece_id=LPDOKJ5RTBNZN6RK2CKYSDWW3V2ZZSSLBS45IMJW63LAFPOIP2DQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.438208746 (2h25m41.219142133s) [noise=true , address=77.90.31.15:4707 , piece_id=6MUCOOI2GWNTJWKMBCMEGGOPABZGTNMJCBRJINLI2DBNH5MDTSRA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.438628794 (2h25m41.219562176s) [noise=true , address=104.36.82.52:4110 , piece_id=2L3FU2G2XJ7KXWKZQFMMFKNUTGFDDIVX6WZ7JDA7ZGR27TUY5XFQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.444249530 (2h25m41.22518291s) [noise=true , address=79.127.205.233:4633 , piece_id=5ZH4CAIECCGSIUHN6SUURPMJDBZJHXZEM5Y6TUCQ6CGLFVLIGQ5A, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.512169007 (2h25m41.293102371s) [upload=2236]: Next returned nil segment Jul 27 00:17:25.512193731 (2h25m41.293127093s) [upload=2236]: Waiting for error group managing segments... Jul 27 00:17:25.538005020 (2h25m41.318938397s) [noise=true , address=77.90.31.15:4707 , piece_id=6MUCOOI2GWNTJWKMBCMEGGOPABZGTNMJCBRJINLI2DBNH5MDTSRA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:25.565397624 (2h25m41.34633101s) [noise=true , address=79.127.220.101:4041 , piece_id=V2HCCLSBVIJ3EHKLGZQTUG2NBKLGXBTLWASNU2YTAUFJLMEJTC2Q, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:25.576511888 (2h25m41.357445269s) [noise=true , address=156.146.43.225:4448 , piece_id=FOD34AK2IRA54YXSIQ677YDARBMQZEVXCVYXV4O54VRTLSXEGXQQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:25.587314374 (2h25m41.368247752s) [noise=true , address=79.127.201.214:4289 , piece_id=3ZTNGA4JFHXXPA4SNPSDTSE4A7BO2XEDYDPDC3FYH3DPGBT5RDYA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:26.035522735 (2h25m41.816456143s) [noise=true , address=79.127.205.235:4669 , piece_id=OYG3OI5WNHYTGHNXRC5XWAZ7K5HXMMPXHE2FCG5WJEYBESEPHP3Q, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:26.171465350 (2h25m41.952398727s) [noise=true , address=79.127.205.246:4280 , piece_id=U2NWGWBUZ56YBIX6JCS4WWAWPJKQL4H6U4OEJLMRGLYMXRPFCDLA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:26.353367125 (2h25m42.134300514s) [noise=true , address=79.127.205.233:4633 , piece_id=5ZH4CAIECCGSIUHN6SUURPMJDBZJHXZEM5Y6TUCQ6CGLFVLIGQ5A, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:28.875651463 (2h25m44.656584832s) [noise=true , address=79.127.219.47:4048 , piece_id=3IRM2Z4VJDDUUUNS55IEJ6QZDL3I7Z2VVXNME7KTK7BJ7KX2ZPAA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:34.668860712 (2h25m50.449794094s) [noise=true , address=45.91.116.14:4206 , piece_id=SJ242HH354E2I64RMLODFM5ILO4JEIQJ533LESMFDMO43QC6SYZQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:35.746949652 (2h25m51.527883016s) [noise=true , address=45.91.116.10:4093 , piece_id=LRNUFEZLW3THSKOTCIRUM2KMAYOEFCZWQG5HIMRAN7LQ3ZBYYPWQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:36.597549852 (2h25m52.378483231s) [noise=true , address=64.49.13.154:4927 , piece_id=4EU77ZGSLMYQL3IW4EITP6HNOKCTV34UE6IXYZQAAGLGGGTSYMHA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:58.869242533 (2h26m14.6501759s) [noise=true , address=79.127.201.215:4360 , piece_id=LPDOKJ5RTBNZN6RK2CKYSDWW3V2ZZSSLBS45IMJW63LAFPOIP2DQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:18:25.363436895 (2h26m41.144370271s) [noise=true , address=104.36.82.56:4252 , piece_id=4C2CKS62DVOK4FD7ANAYPFSWA44RPIFCPAQ2JMSKGBCQVWBDT4BA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1UDFNiqXb4riRuewJDPrKFykVWLiryNKqgkH6uCZszz6Sr8naB, address:104.36.82.56:4252): protocol: rpc: dial tcp 104.36.82.56:4252: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363499012 (2h26m41.144432382s) [noise=true , address=104.36.82.52:4110 , piece_id=2L3FU2G2XJ7KXWKZQFMMFKNUTGFDDIVX6WZ7JDA7ZGR27TUY5XFQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363574402 (2h26m41.144507767s) [noise=true , address=104.36.82.53:4121 , piece_id=5NX3GTJG4GVSAD3ELLCHAF2O2AKIZS5OLCYWFI2WXAYYEYXQB4VQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: write tcp 159.223.143.39:56734->104.36.82.53:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363634515 (2h26m41.144567878s) [noise=true , address=104.36.82.47:4540 , piece_id=IKVWDCQM7PCG3BK4IFGLEZTHVUTFNOLYGWJCONO7OJGO27KXGWGQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:104.36.82.47:4540): protocol: rpc: dial tcp 104.36.82.47:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363660665 (2h26m41.144594041s) [noise=true , address=104.36.82.46:4486 , piece_id=IRF7MBR3UJ2I66DUAE7WRDYIXTT4KN3B744BR3ZSD5M6KMNX2YGQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic, address:104.36.82.46:4486): protocol: rpc: dial tcp 104.36.82.46:4486: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363722746 (2h26m41.144656122s) [noise=true , address=104.36.82.54:4169 , piece_id=P5K5LAJSGGIG45SCX5P74A7ZH6BBKPRO26TFVF7I523LOWWOCSRQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363756447 (2h26m41.144689839s) [seg_pos={0 0}, upload=2236]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:18:25.391608850 (2h26m41.172542217s) [upload=2236]: Done uploading segments... Jul 27 00:18:25.391646216 (2h26m41.172579581s) [upload=2236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.030876395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X.] duration: 1.94573129s FAIL (2025-07-27 00:18:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.9457545s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X] duration: 30.027866191s FAIL (2025-07-27 00:18:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027893431s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.XXX...............X..X......X.XX....X...X.............] duration: 1.982682937s FAIL (2025-07-27 00:18:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.982708187s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.XX..............X.X........X...XX..........................] duration: 37.322683533s FAIL (2025-07-27 00:17:46 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.322715592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031465588s FAIL (2025-07-27 00:17:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m41.066136758s) [upload=2221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:16:25.285209159 (2h24m41.066142533s) [upload=2221]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:16:25.319767806 (2h24m41.100701192s) [seg_pos={0 0}, upload=2221]: Begin upload segment... Jul 27 00:16:25.319831084 (2h24m41.100764457s) [seg_pos={0 0}, upload=2221]: Done begin upload segment. Jul 27 00:16:25.319865681 (2h24m41.100799063s) [noise=true , address=104.36.82.47:4525 , piece_id=SYVPP53DTYQ5K45AZKJZTA3BIB5ANJWLCZYHDZ3JBCLXPBREHMQQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.320389233 (2h24m41.101322621s) [noise=true , address=79.127.205.235:4685 , piece_id=J2NJBTM7GUEKBWQOBIWCR3V5AU6Q32VFZJTMBN2LOLRYEQFVSRBQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.320626047 (2h24m41.101559427s) [noise=true , address=64.49.13.154:4927 , piece_id=UUTIFM5HZIJCFUXN6SL4KR2ZQWZMSTDAZXD4R6E4IYDMA2TM2OIA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.320981090 (2h24m41.10191447s) [noise=true , address=104.36.82.5:4870 , piece_id=GPGSASZ3DHWYK3TFROF7YPESLPCCEVOQCCJXCR62VPUIELHNGHLQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.321066740 (2h24m41.102000126s) [noise=true , address=64.49.13.138:4482 , piece_id=P7ICQS4NSIEDV7DDVMAT2MQTMDV6HL3D4O4UUJR4NZGP6QB73OKA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.321572572 (2h24m41.102505959s) [noise=true , address=156.146.43.225:4434 , piece_id=4OWXBZ2WOK44KLD7QSSEPPA2AS3HUULUPTRTRPZ5636GXEAFTBFA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.321764012 (2h24m41.102697399s) [noise=true , address=79.127.220.98:4422 , piece_id=5MTPUQJFF2PT76S4QIAOACBARGZWVWV4RNBZR7I6SRDH5BQW66JA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322124162 (2h24m41.103057542s) [noise=true , address=79.127.205.234:4618 , piece_id=RXXCZKZBRZTT7M5BJA3D7RYOOPZBY6CPE2SL3P22MBX5RGMW6ALA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322142587 (2h24m41.103075972s) [noise=true , address=64.49.13.139:4478 , piece_id=V2VZ7OBAWPRTL4BRGHVKMLQ6LNMKGGAFJLY42ZPP5LNHS2MILFYA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322451838 (2h24m41.103385228s) [noise=true , address=79.127.219.47:4048 , piece_id=T63KXMHJDL5YD6QMNEKX5SGJRGT6ET7FNH67ZJ4RJ55IN6PPIHIA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322583430 (2h24m41.103516807s) [noise=true , address=79.127.205.233:4623 , piece_id=5RVMBBL6M77NOHVCDF3BQB7MJXF23QDWS2C4GI6E23CTXLWF77AA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323110073 (2h24m41.104043455s) [noise=true , address=79.127.201.215:4343 , piece_id=7NDNNBLR5GHIFYYGJCDWFTGB2FPUOOUBIPQO5UO7GEHKMXSCQW6A, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323166956 (2h24m41.104100336s) [noise=true , address=104.36.82.46:4506 , piece_id=KY63MZ3U5CHALKJOSYHEXG3SAHMG4Z5WHZ5IYJZXG54UALSF77NQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323565464 (2h24m41.104498848s) [noise=true , address=79.127.205.245:4225 , piece_id=IFZEESA7RZJPNFKMEJXHLUPQNBTRTBZC4RF2HC7QPGR5C54GCJ6Q, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323762115 (2h24m41.104695494s) [noise=true , address=45.91.116.14:4206 , piece_id=CUSXUDHNMSNVHDS7GZETEFYQOUHEGEOUZQXADSZRNPKE4Z6AJMJA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323889183 (2h24m41.104822563s) [noise=true , address=104.36.82.55:4215 , piece_id=ZEDZFH2P2R2UUSNOSUK3N2W6WIWX64KKY3YOCS6GJW5YDZ3K44EQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.324445314 (2h24m41.105378702s) [noise=true , address=77.90.31.18:4833 , piece_id=QX7KRNGVZDHHQ2WU3RBFRZLRI3LAC4HFV75MDNNWNFZT2344J25A, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.324637924 (2h24m41.10557131s) [noise=true , address=104.36.82.54:4176 , piece_id=RUB2MGBLQUVJFR3YAU3BTFHJXT5SITU5GYXD4RSQF5KTAVP3HXKQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.405885455 (2h24m41.186818824s) [upload=2221]: Next returned nil segment Jul 27 00:16:25.405915264 (2h24m41.186848646s) [upload=2221]: Waiting for error group managing segments... Jul 27 00:16:25.738834051 (2h24m41.519767419s) [noise=true , address=156.146.43.225:4434 , piece_id=4OWXBZ2WOK44KLD7QSSEPPA2AS3HUULUPTRTRPZ5636GXEAFTBFA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.925164489 (2h24m41.70609785s) [noise=true , address=79.127.205.235:4685 , piece_id=J2NJBTM7GUEKBWQOBIWCR3V5AU6Q32VFZJTMBN2LOLRYEQFVSRBQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.925277473 (2h24m41.706210837s) [noise=true , address=64.49.13.138:4482 , piece_id=P7ICQS4NSIEDV7DDVMAT2MQTMDV6HL3D4O4UUJR4NZGP6QB73OKA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 27 00:16:25.932252393 (2h24m41.713185754s) [noise=true , address=79.127.205.234:4618 , piece_id=RXXCZKZBRZTT7M5BJA3D7RYOOPZBY6CPE2SL3P22MBX5RGMW6ALA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.936905485 (2h24m41.71783885s) [noise=true , address=79.127.205.233:4623 , piece_id=5RVMBBL6M77NOHVCDF3BQB7MJXF23QDWS2C4GI6E23CTXLWF77AA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.937811783 (2h24m41.718745151s) [noise=true , address=79.127.205.245:4225 , piece_id=IFZEESA7RZJPNFKMEJXHLUPQNBTRTBZC4RF2HC7QPGR5C54GCJ6Q, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:26.249105509 (2h24m42.03003888s) [noise=true , address=45.91.116.14:4206 , piece_id=CUSXUDHNMSNVHDS7GZETEFYQOUHEGEOUZQXADSZRNPKE4Z6AJMJA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:27.056267585 (2h24m42.837200967s) [noise=true , address=64.49.13.139:4478 , piece_id=V2VZ7OBAWPRTL4BRGHVKMLQ6LNMKGGAFJLY42ZPP5LNHS2MILFYA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 27 00:16:27.847906157 (2h24m43.628839525s) [noise=true , address=79.127.219.47:4048 , piece_id=T63KXMHJDL5YD6QMNEKX5SGJRGT6ET7FNH67ZJ4RJ55IN6PPIHIA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:32.415161600 (2h24m48.196094969s) [noise=true , address=77.90.31.18:4833 , piece_id=QX7KRNGVZDHHQ2WU3RBFRZLRI3LAC4HFV75MDNNWNFZT2344J25A, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:33.390197542 (2h24m49.171130906s) [noise=true , address=79.127.220.98:4422 , piece_id=5MTPUQJFF2PT76S4QIAOACBARGZWVWV4RNBZR7I6SRDH5BQW66JA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:35.324489552 (2h24m51.105422921s) [noise=true , address=64.49.13.154:4927 , piece_id=UUTIFM5HZIJCFUXN6SL4KR2ZQWZMSTDAZXD4R6E4IYDMA2TM2OIA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:38.959115923 (2h24m54.740049292s) [noise=true , address=79.127.201.215:4343 , piece_id=7NDNNBLR5GHIFYYGJCDWFTGB2FPUOOUBIPQO5UO7GEHKMXSCQW6A, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:17:25.256581505 (2h25m41.03751488s) [noise=true , address=104.36.82.46:4506 , piece_id=KY63MZ3U5CHALKJOSYHEXG3SAHMG4Z5WHZ5IYJZXG54UALSF77NQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: rpc: dial tcp 104.36.82.46:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256643185 (2h25m41.037576566s) [noise=true , address=104.36.82.54:4176 , piece_id=RUB2MGBLQUVJFR3YAU3BTFHJXT5SITU5GYXD4RSQF5KTAVP3HXKQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256722308 (2h25m41.037655674s) [noise=true , address=104.36.82.47:4525 , piece_id=SYVPP53DTYQ5K45AZKJZTA3BIB5ANJWLCZYHDZ3JBCLXPBREHMQQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: rpc: dial tcp 104.36.82.47:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256804080 (2h25m41.037737459s) [noise=true , address=104.36.82.5:4870 , piece_id=GPGSASZ3DHWYK3TFROF7YPESLPCCEVOQCCJXCR62VPUIELHNGHLQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1MmdhnDp6kKktFb5btZUDeVQ2u3CNUKCZU7RfuEqiSQ2TEqwKX, address:104.36.82.5:4870): protocol: rpc: dial tcp 104.36.82.5:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256876115 (2h25m41.037809494s) [noise=true , address=104.36.82.55:4215 , piece_id=ZEDZFH2P2R2UUSNOSUK3N2W6WIWX64KKY3YOCS6GJW5YDZ3K44EQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:49220->104.36.82.55:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256931590 (2h25m41.037864972s) [seg_pos={0 0}, upload=2221]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:17:25.285948243 (2h25m41.06688162s) [upload=2221]: Done uploading segments... Jul 27 00:17:25.285977783 (2h25m41.066911162s) [upload=2221]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.031482318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.XXX...............X..X......X.XX....X...X............] duration: 1.841856237s FAIL (2025-07-27 00:17:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.841889508s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X..] duration: 1.213587711s FAIL (2025-07-27 00:17:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.213630039s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....X.XXX...............X..X......X.XX....X...X...........] duration: 1.92336075s FAIL (2025-07-27 00:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.923382281s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.] duration: 536.469461ms FAIL (2025-07-27 00:16:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.501926ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..XX..............X.X........X...XX.........................] duration: 13.239966644s FAIL (2025-07-27 00:16:36 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.239995807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027783341s FAIL (2025-07-27 00:16:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 877697s) [upload=2206]: Got next segment. Inline: false Jul 27 00:15:25.204124474 (2h23m40.985057875s) [upload=2206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:25.204133499 (2h23m40.985066888s) [upload=2206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:25.246103485 (2h23m41.027036869s) [seg_pos={0 0}, upload=2206]: Begin upload segment... Jul 27 00:15:25.246165900 (2h23m41.027099266s) [seg_pos={0 0}, upload=2206]: Done begin upload segment. Jul 27 00:15:25.246193432 (2h23m41.02712681s) [noise=true , address=79.127.205.245:4240 , piece_id=RTEASNNBTDSX7XTLRQZJOYHCLI47SUQBKVABIXC5BHVYIGFGZ63A, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.246622239 (2h23m41.027555625s) [noise=true , address=79.127.205.235:4690 , piece_id=WJI6X4YMEM3RIYSNFZAAGRLP2BQNO3Z37GPM7E4VBKNRLAK7LUWQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.246962606 (2h23m41.027895986s) [noise=true , address=104.36.82.54:4166 , piece_id=P7ZZSYJMEKFEEPXPNPMB5PYOWMRVNNUH3KEFH5F7CNGZ4QMH3KOQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247172901 (2h23m41.028106282s) [noise=true , address=79.127.219.38:4001 , piece_id=JFTNZ727CPV66POZJXI4NK7TBDZG3OR3YXTVT3SGO3JZV7VKU2NQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247301713 (2h23m41.028235094s) [noise=true , address=77.90.31.15:4707 , piece_id=MDPCCZJW7EZZDJ7YWXD27F3H57JJJ3275HZ4EUWGOWOBPCNIYKNQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247616338 (2h23m41.02854974s) [noise=true , address=79.127.201.215:4359 , piece_id=SBYNPEMOYTGTAVUO6WILDNJRDXRKIV6PNM52NEUPUDZS3GQAC4LQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247662824 (2h23m41.028596202s) [noise=true , address=77.90.31.10:4531 , piece_id=ZBSKCZZP6GCYJSKC5FUN5SYZSGC4JSHJBP6U2FAOBJOKIV2Z4UNQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.248173859 (2h23m41.02910724s) [noise=true , address=79.127.220.97:4361 , piece_id=SKOHDTJH63XK3VQSTFOYKE7FUN6SK76KRPL4ZJJ7VTX55DNN5TNQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.248568305 (2h23m41.029501686s) [noise=true , address=156.146.43.225:4434 , piece_id=R2OKXNV47534RXMOE5YDLA3OBIK5DIHWYILJDQWSURZLPGZFQ3ZQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.248988972 (2h23m41.029922352s) [noise=true , address=77.90.31.12:4616 , piece_id=YO7377KIVG6GKATASD7NWZVY7NA2AJPQE6KD2VTXQ46MVXVDYQ7A, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.249443834 (2h23m41.030377212s) [noise=true , address=104.36.82.47:4545 , piece_id=RP6FNM4QTH5Z5MJO2FDPWLI7KLKOLV2WXTIM4MM2JFWDFSN7XCCA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.249851997 (2h23m41.030785381s) [noise=true , address=64.49.13.140:4508 , piece_id=LVHMPRDJNA6TXGOJSNO7YQIJU2J4U3E2OANN7JUYE3GQC6Y23BIQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.250276585 (2h23m41.031209967s) [noise=true , address=104.36.82.53:4128 , piece_id=XL6U67NQKSCBYJ3M5VC5XZ5CT43LTLILRZ2XQQLNUXKAI2QKD2JA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.250717679 (2h23m41.031651062s) [noise=true , address=79.127.201.214:4321 , piece_id=BLSZTKBBI6EPQZJBTKCYA2Q2NFBOVIMUHMFZYGBIG2ELZ2W5RERA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.251568787 (2h23m41.032502169s) [noise=true , address=64.49.13.154:4927 , piece_id=DPT2QCUKTUW7HUQCZTONGC4AZVPR54UROP4PHZBBQS2WJCQJUTXA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.252833752 (2h23m41.033767134s) [noise=true , address=79.127.220.101:4041 , piece_id=C4U4ID7UKFLZSZUTCLLKEDDV63BZOXB5AAPWXVTNGOIEWS3EMNLQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.253154113 (2h23m41.034087491s) [noise=true , address=79.127.205.234:4618 , piece_id=EIX3FPJ3PO6EAAEFYYSVQFB5E3G2YUFGFP6XZ7GEGWHSVWKL3O2Q, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.253411742 (2h23m41.03434512s) [noise=true , address=77.90.31.21:4896 , piece_id=CRMTZHBF7D7S46KFGNDS676EAP3EOVF5NUH6RTYFUY4C62TKM2UQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.317217016 (2h23m41.098150412s) [upload=2206]: Next returned nil segment Jul 27 00:15:25.317238420 (2h23m41.098171804s) [upload=2206]: Waiting for error group managing segments... Jul 27 00:15:25.323642403 (2h23m41.104575783s) [noise=true , address=79.127.219.38:4001 , piece_id=JFTNZ727CPV66POZJXI4NK7TBDZG3OR3YXTVT3SGO3JZV7VKU2NQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.352837629 (2h23m41.133770999s) [noise=true , address=77.90.31.15:4707 , piece_id=MDPCCZJW7EZZDJ7YWXD27F3H57JJJ3275HZ4EUWGOWOBPCNIYKNQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.833445711 (2h23m41.614379088s) [noise=true , address=79.127.205.245:4240 , piece_id=RTEASNNBTDSX7XTLRQZJOYHCLI47SUQBKVABIXC5BHVYIGFGZ63A, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.903497841 (2h23m41.68443121s) [noise=true , address=79.127.205.235:4690 , piece_id=WJI6X4YMEM3RIYSNFZAAGRLP2BQNO3Z37GPM7E4VBKNRLAK7LUWQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.979282690 (2h23m41.760216077s) [noise=true , address=79.127.205.234:4618 , piece_id=EIX3FPJ3PO6EAAEFYYSVQFB5E3G2YUFGFP6XZ7GEGWHSVWKL3O2Q, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:26.176335879 (2h23m41.957269258s) [noise=true , address=64.49.13.154:4927 , piece_id=DPT2QCUKTUW7HUQCZTONGC4AZVPR54UROP4PHZBBQS2WJCQJUTXA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:26.266294508 (2h23m42.047227889s) [noise=true , address=79.127.220.97:4361 , piece_id=SKOHDTJH63XK3VQSTFOYKE7FUN6SK76KRPL4ZJJ7VTX55DNN5TNQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:26.353497230 (2h23m42.13443061s) [noise=true , address=64.49.13.140:4508 , piece_id=LVHMPRDJNA6TXGOJSNO7YQIJU2J4U3E2OANN7JUYE3GQC6Y23BIQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:15:29.161225666 (2h23m44.942159057s) [noise=true , address=79.127.201.214:4321 , piece_id=BLSZTKBBI6EPQZJBTKCYA2Q2NFBOVIMUHMFZYGBIG2ELZ2W5RERA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:29.667738457 (2h23m45.44867183s) [noise=true , address=79.127.220.101:4041 , piece_id=C4U4ID7UKFLZSZUTCLLKEDDV63BZOXB5AAPWXVTNGOIEWS3EMNLQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:30.515418509 (2h23m46.29635188s) [noise=true , address=156.146.43.225:4434 , piece_id=R2OKXNV47534RXMOE5YDLA3OBIK5DIHWYILJDQWSURZLPGZFQ3ZQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:45.149308791 (2h24m0.93024217s) [noise=true , address=77.90.31.12:4616 , piece_id=YO7377KIVG6GKATASD7NWZVY7NA2AJPQE6KD2VTXQ46MVXVDYQ7A, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:45.332301191 (2h24m1.113234552s) [noise=true , address=79.127.201.215:4359 , piece_id=SBYNPEMOYTGTAVUO6WILDNJRDXRKIV6PNM52NEUPUDZS3GQAC4LQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:16:25.172775922 (2h24m40.953709307s) [noise=true , address=104.36.82.53:4128 , piece_id=XL6U67NQKSCBYJ3M5VC5XZ5CT43LTLILRZ2XQQLNUXKAI2QKD2JA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 159.223.143.39:34196->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.172935510 (2h24m40.953868887s) [noise=true , address=104.36.82.47:4545 , piece_id=RP6FNM4QTH5Z5MJO2FDPWLI7KLKOLV2WXTIM4MM2JFWDFSN7XCCA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173018375 (2h24m40.953951752s) [noise=true , address=77.90.31.21:4896 , piece_id=CRMTZHBF7D7S46KFGNDS676EAP3EOVF5NUH6RTYFUY4C62TKM2UQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173074734 (2h24m40.954008095s) [noise=true , address=77.90.31.10:4531 , piece_id=ZBSKCZZP6GCYJSKC5FUN5SYZSGC4JSHJBP6U2FAOBJOKIV2Z4UNQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: write tcp 159.223.143.39:46976->77.90.31.10:4531: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173092578 (2h24m40.954025958s) [noise=true , address=104.36.82.54:4166 , piece_id=P7ZZSYJMEKFEEPXPNPMB5PYOWMRVNNUH3KEFH5F7CNGZ4QMH3KOQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 159.223.143.39:44726->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173130344 (2h24m40.954063724s) [seg_pos={0 0}, upload=2206]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:16:25.198981863 (2h24m40.979915235s) [upload=2206]: Done uploading segments... Jul 27 00:16:25.199011535 (2h24m40.979944916s) [upload=2206]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.027808506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.] duration: 462.926429ms FAIL (2025-07-27 00:16:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.962737ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X] duration: 30.034506899s FAIL (2025-07-27 00:16:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034532237s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...XX..............X.X........X...XX........................] duration: 39.516002531s FAIL (2025-07-27 00:15:55 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.516028424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.....X.XXX...............X..X......X.XX....X...X..........] duration: 1.937379626s FAIL (2025-07-27 00:15:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.937431686s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X] duration: 30.026313839s FAIL (2025-07-27 00:15:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026336629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...] duration: 1.837549174s FAIL (2025-07-27 00:15:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.837601677s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027212135s FAIL (2025-07-27 00:15:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 25.114292822 (2h22m40.895226199s) [upload=2189]: Uploading segments... Jul 27 00:14:25.114499427 (2h22m40.895432812s) [upload=2189]: Got next segment. Inline: false Jul 27 00:14:25.114563626 (2h22m40.895496989s) [upload=2189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:14:25.114569609 (2h22m40.895502972s) [upload=2189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:14:25.150755361 (2h22m40.931688751s) [seg_pos={0 0}, upload=2189]: Begin upload segment... Jul 27 00:14:25.150820002 (2h22m40.931753367s) [seg_pos={0 0}, upload=2189]: Done begin upload segment. Jul 27 00:14:25.150844952 (2h22m40.93177833s) [noise=true , address=77.90.31.15:4714 , piece_id=GER5UP2KHUDC6OTJVQF7F2MAO6R4L3QRM5B7YEFEJKLRLBGXFMJA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.151095640 (2h22m40.932029008s) [noise=true , address=45.91.116.9:4277 , piece_id=RYTK6HJWKERZ6LCEL2ML4KOEFKYGKMVPUJMJ2U5H52CEHEFBPKYQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.151491664 (2h22m40.932425033s) [noise=true , address=104.36.82.47:4520 , piece_id=XSPVKBJQP3F7YOD5PLKD7TT26ITLFP4XGAAUYI7M2JLYSTL7BIQA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.155895216 (2h22m40.936828584s) [noise=true , address=77.90.31.16:4767 , piece_id=E2GI52NZ4LYOD4OPLGIRXROP6OHSPZSF2NAL4LU4FO664S6GPH6Q, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.156369303 (2h22m40.937302671s) [noise=true , address=104.36.82.52:4085 , piece_id=XB2XNBIDJYWEN3L5SOYDCNCTNOSAF6QBTNHNB3KFOYXIGOVK67CA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.156693469 (2h22m40.937626838s) [noise=true , address=104.36.82.56:4256 , piece_id=CJS7S3GFCJ7CPKEGFLKYZ3NKFZ5VA5S5RVXEYSAF5GFSZGLED24A, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.157084137 (2h22m40.938017507s) [noise=true , address=79.127.220.101:4048 , piece_id=5UY4WS2KCXEVKJTTF4OPQVOXHCSHWRO4A42NDLSZ24763PAF5H4A, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.157474635 (2h22m40.938408011s) [noise=true , address=79.127.205.245:4251 , piece_id=3GX6BDN2WUR4O6SOS5DWM5SPP2JVW2QCCC3RDWDA3OQEEA3JIEYQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.157815005 (2h22m40.938748374s) [noise=true , address=79.127.205.235:4685 , piece_id=YAWDYVM6GCHFREGFF4SQQOEM5PDR3EZ5LYHER6373C452KLMT5QA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.158269401 (2h22m40.939202765s) [noise=true , address=156.146.43.225:4448 , piece_id=YRKDOTWU4PPVCVF7G4N2O66CNKGCQZ3XTVXIHCONDWIDT2S3EAMQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.159559710 (2h22m40.940493078s) [noise=true , address=79.127.219.38:4010 , piece_id=R65EBXZP6IKVZPGMERDQWQSW2SAOMTCTP5JER55QAVGAFZUXIT6A, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.160620612 (2h22m40.941553973s) [noise=true , address=77.90.31.21:4906 , piece_id=AM7UFHAGRMVTG4PS63S4TSLHD5BZO6JGQOFO653HBDVOOHIKIABQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.163008733 (2h22m40.9439421s) [noise=true , address=156.146.43.225:4449 , piece_id=B2IFELEFV5465S2OHRIIUZ54BINQYYBB7LW365DYTKZEMAKEXWZQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.165127857 (2h22m40.946061225s) [noise=true , address=64.49.13.138:4482 , piece_id=SBUMXCEGX4HB62UQ64NTMUES4D2CEC5KD67WZET2PHN456FDPQMA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.167229434 (2h22m40.948162803s) [noise=true , address=79.127.205.246:4282 , piece_id=23LMGBHFCXME4YKM3BTDH5OQQMLAUZOKDUWXHMRSBYOV2AIHOVFQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.169740413 (2h22m40.950673784s) [noise=true , address=79.127.220.97:4361 , piece_id=42ZIAHK6OMVYS5JDHQRV5VRPUG2KGW6PJ4PZBUW5WHPR36HVCWLA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.171926988 (2h22m40.952860357s) [noise=true , address=104.36.82.54:4178 , piece_id=EIF5VSO7EIRHF57BJ3EO2ALCNK5FWGHC56MQNJ3MHTXQ6LTOTRWQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.174056138 (2h22m40.954989507s) [noise=true , address=79.127.201.214:4289 , piece_id=UFPTMNI7QEZ3KVR2IWNAS4GYW7MVK72QYBUANOEH5OS7U5ENIKZA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.233441813 (2h22m41.014375175s) [upload=2189]: Next returned nil segment Jul 27 00:14:25.233464793 (2h22m41.014398156s) [upload=2189]: Waiting for error group managing segments... Jul 27 00:14:25.253316208 (2h22m41.034249571s) [noise=true , address=79.127.219.38:4010 , piece_id=R65EBXZP6IKVZPGMERDQWQSW2SAOMTCTP5JER55QAVGAFZUXIT6A, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.283253681 (2h22m41.064187054s) [noise=true , address=77.90.31.15:4714 , piece_id=GER5UP2KHUDC6OTJVQF7F2MAO6R4L3QRM5B7YEFEJKLRLBGXFMJA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.301891412 (2h22m41.08282479s) [noise=true , address=77.90.31.16:4767 , piece_id=E2GI52NZ4LYOD4OPLGIRXROP6OHSPZSF2NAL4LU4FO664S6GPH6Q, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.662617715 (2h22m41.443551081s) [noise=true , address=79.127.220.101:4048 , piece_id=5UY4WS2KCXEVKJTTF4OPQVOXHCSHWRO4A42NDLSZ24763PAF5H4A, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:14:25.766214396 (2h22m41.547147765s) [noise=true , address=79.127.205.246:4282 , piece_id=23LMGBHFCXME4YKM3BTDH5OQQMLAUZOKDUWXHMRSBYOV2AIHOVFQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.831904084 (2h22m41.612837445s) [noise=true , address=79.127.205.245:4251 , piece_id=3GX6BDN2WUR4O6SOS5DWM5SPP2JVW2QCCC3RDWDA3OQEEA3JIEYQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:26.008447440 (2h22m41.789380824s) [noise=true , address=79.127.205.235:4685 , piece_id=YAWDYVM6GCHFREGFF4SQQOEM5PDR3EZ5LYHER6373C452KLMT5QA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:28.219396960 (2h22m44.000330336s) [noise=true , address=79.127.220.97:4361 , piece_id=42ZIAHK6OMVYS5JDHQRV5VRPUG2KGW6PJ4PZBUW5WHPR36HVCWLA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:29.654928227 (2h22m45.435861596s) [noise=true , address=156.146.43.225:4449 , piece_id=B2IFELEFV5465S2OHRIIUZ54BINQYYBB7LW365DYTKZEMAKEXWZQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:30.560049952 (2h22m46.340983321s) [noise=true , address=156.146.43.225:4448 , piece_id=YRKDOTWU4PPVCVF7G4N2O66CNKGCQZ3XTVXIHCONDWIDT2S3EAMQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:31.339084597 (2h22m47.120017973s) [noise=true , address=45.91.116.9:4277 , piece_id=RYTK6HJWKERZ6LCEL2ML4KOEFKYGKMVPUJMJ2U5H52CEHEFBPKYQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:14:33.405127699 (2h22m49.186061067s) [noise=true , address=79.127.201.214:4289 , piece_id=UFPTMNI7QEZ3KVR2IWNAS4GYW7MVK72QYBUANOEH5OS7U5ENIKZA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:35.816824455 (2h22m51.597757822s) [noise=true , address=64.49.13.138:4482 , piece_id=SBUMXCEGX4HB62UQ64NTMUES4D2CEC5KD67WZET2PHN456FDPQMA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:15:25.077942899 (2h23m40.858876279s) [noise=true , address=77.90.31.21:4906 , piece_id=AM7UFHAGRMVTG4PS63S4TSLHD5BZO6JGQOFO653HBDVOOHIKIABQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:25.077991961 (2h23m40.858925337s) [noise=true , address=104.36.82.56:4256 , piece_id=CJS7S3GFCJ7CPKEGFLKYZ3NKFZ5VA5S5RVXEYSAF5GFSZGLED24A, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:25.078128958 (2h23m40.85906234s) [noise=true , address=104.36.82.54:4178 , piece_id=EIF5VSO7EIRHF57BJ3EO2ALCNK5FWGHC56MQNJ3MHTXQ6LTOTRWQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:104.36.82.54:4178): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:15:25.078259537 (2h23m40.859192912s) [noise=true , address=104.36.82.47:4520 , piece_id=XSPVKBJQP3F7YOD5PLKD7TT26ITLFP4XGAAUYI7M2JLYSTL7BIQA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:15:25.078446789 (2h23m40.859380154s) [noise=true , address=104.36.82.52:4085 , piece_id=XB2XNBIDJYWEN3L5SOYDCNCTNOSAF6QBTNHNB3KFOYXIGOVK67CA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: rpc: dial tcp 104.36.82.52:4085: i/o timeout; conn closed: op Jul 27 00:15:25.078478910 (2h23m40.859412285s) [seg_pos={0 0}, upload=2189]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:15:25.101754143 (2h23m40.882687512s) [upload=2189]: Done uploading segments... Jul 27 00:15:25.101797259 (2h23m40.882730635s) [upload=2189]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.02724214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X..] duration: 775.871515ms FAIL (2025-07-27 00:15:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 775.905699ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX.....X.XXX...............X..X......X.XX....X...X.........] duration: 1.982151706s FAIL (2025-07-27 00:14:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.982179014s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...] duration: 1.783408113s FAIL (2025-07-27 00:14:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.783468183s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....XX..............X.X........X...XX.......................] duration: 14.468555081s FAIL (2025-07-27 00:14:26 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.468596088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X..] duration: 1.912088458s FAIL (2025-07-27 00:14:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.912158471s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.024966463s FAIL (2025-07-27 00:14:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.611174432 (2h21m27.392107833s) [noise=true , address=104.36.82.55:4216 , piece_id=B2QBMDBSB2CR5XL2B4M7JGG3YFUD2EWOFV5VVKBDNLKW3ON375NA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.611503399 (2h21m27.392436789s) [noise=true , address=77.90.31.16:4746 , piece_id=SDHOWJHHDCHWXZJV5VEHJCJYU5ETVIEXWAQAEUO4RZRZJPRDNJDQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.611675073 (2h21m27.39260846s) [noise=true , address=156.146.43.225:4448 , piece_id=KBIZ5QKQOLU7F3JBDCYWFXP5NEG73XK5L6A5HXMILKXSX42ZNW7Q, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612101828 (2h21m27.393035217s) [noise=true , address=79.127.205.245:4250 , piece_id=ODIMS6ZUNZMF5V6NK7JBLTRXMUXOL2C2TYV4LVW7OAJBKSTLYORQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612190737 (2h21m27.393124123s) [noise=true , address=104.36.82.52:4104 , piece_id=XUVOXJCDWGRWDCCO2LNGIX24BROQKDOVJJD4RP4PEO26ITMGLEPA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612615908 (2h21m27.393549296s) [noise=true , address=79.127.201.215:4357 , piece_id=4GRR477U3XQ65QMONSBESNRPVWSX3RRRLG5JRVU77BN33D4FPZLQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612902304 (2h21m27.393835691s) [noise=true , address=79.127.201.214:4321 , piece_id=VJJQ6PY7PZM76IU4DQ2BU5EEFG4QGLCZRYAWOAAX7XCNKKXTMRVA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.613018061 (2h21m27.393951447s) [noise=true , address=104.36.82.2:4740 , piece_id=2CK7IGYFNQN4DEGG5YNHZ6B4ECVCYZX44AEV72ZAQCLWK2F2MFOA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.613452167 (2h21m27.394385556s) [noise=true , address=104.36.82.46:4503 , piece_id=IT7AHX64ZIOOIFFM62EUELXD7VIHIPUQP7E4JP3UVHKGRR7B6FNQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.613905389 (2h21m27.394838815s) [noise=true , address=79.127.201.214:4289 , piece_id=JZZY3QTUDRZYOUQPN6UHB7YZEYYUZCXPXJCRQQLWMHKFKAC3V5LQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614324661 (2h21m27.395258051s) [noise=true , address=64.49.13.26:4953 , piece_id=4US56S2F3JD34WCAZA7AABI6JZHX4ZKF2JB3JMXBCD723XBIQHBQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614479301 (2h21m27.395412692s) [noise=true , address=79.127.205.246:4262 , piece_id=JPRU4ACG7VLQBWB2DH5LIZY5S2QAAH4UYNVKAHFPBGYDRG4ETO3A, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614731327 (2h21m27.395664721s) [noise=true , address=79.127.219.38:4001 , piece_id=ST4QY2BAH32ADFZOXGJBK2AL62HJKZQ7K57CIH6HOU74FSTKW3OQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614977081 (2h21m27.39591047s) [noise=true , address=104.36.82.42:4829 , piece_id=MWASMHZGME3OJDBOFSGPDOPCSRE2R3HXAVI2UUFNYGMQPOJ22D7A, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.615189572 (2h21m27.396122962s) [noise=true , address=77.90.31.18:4821 , piece_id=C33PAKSCFIWYLKC3ZMX7JSKFTEB4TID5IVKDTGHJGHHHCGL2XEYA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.615599924 (2h21m27.396533312s) [noise=true , address=77.90.31.19:4875 , piece_id=F7DUM74CQ7EJENYKTIJDSPD72KLK7RADE7EMQG5O5VJBGE5HN7BQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.718126438 (2h21m27.499059829s) [upload=2170]: Next returned nil segment Jul 27 00:13:11.718148911 (2h21m27.499082299s) [upload=2170]: Waiting for error group managing segments... Jul 27 00:13:11.765404359 (2h21m27.546337736s) [noise=true , address=77.90.31.16:4746 , piece_id=SDHOWJHHDCHWXZJV5VEHJCJYU5ETVIEXWAQAEUO4RZRZJPRDNJDQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:11.766674933 (2h21m27.547608309s) [noise=true , address=156.146.43.225:4448 , piece_id=KBIZ5QKQOLU7F3JBDCYWFXP5NEG73XK5L6A5HXMILKXSX42ZNW7Q, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:11.773175838 (2h21m27.554109218s) [noise=true , address=79.127.201.214:4321 , piece_id=VJJQ6PY7PZM76IU4DQ2BU5EEFG4QGLCZRYAWOAAX7XCNKKXTMRVA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:11.803170204 (2h21m27.58410359s) [noise=true , address=79.127.201.214:4289 , piece_id=JZZY3QTUDRZYOUQPN6UHB7YZEYYUZCXPXJCRQQLWMHKFKAC3V5LQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:11.893244332 (2h21m27.674177702s) [noise=true , address=79.127.219.38:4001 , piece_id=ST4QY2BAH32ADFZOXGJBK2AL62HJKZQ7K57CIH6HOU74FSTKW3OQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:12.197473394 (2h21m27.978406772s) [noise=true , address=79.127.205.245:4250 , piece_id=ODIMS6ZUNZMF5V6NK7JBLTRXMUXOL2C2TYV4LVW7OAJBKSTLYORQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:12.240826317 (2h21m28.0217597s) [noise=true , address=79.127.205.246:4262 , piece_id=JPRU4ACG7VLQBWB2DH5LIZY5S2QAAH4UYNVKAHFPBGYDRG4ETO3A, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:12.726449912 (2h21m28.507383275s) [noise=true , address=79.127.205.233:4650 , piece_id=MRO5FS7MCY7SVNDT57VAAQ6WEIDUZIDJGUVS2DWV7LCI76OZOC4Q, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:14.349472359 (2h21m30.130405736s) [noise=true , address=64.49.13.26:4953 , piece_id=4US56S2F3JD34WCAZA7AABI6JZHX4ZKF2JB3JMXBCD723XBIQHBQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:19.893805564 (2h21m35.674738943s) [noise=true , address=79.127.201.215:4357 , piece_id=4GRR477U3XQ65QMONSBESNRPVWSX3RRRLG5JRVU77BN33D4FPZLQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:29.039662290 (2h21m44.820595666s) [noise=true , address=77.90.31.19:4875 , piece_id=F7DUM74CQ7EJENYKTIJDSPD72KLK7RADE7EMQG5O5VJBGE5HN7BQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:14:11.536688272 (2h22m27.317621668s) [noise=true , address=104.36.82.46:4503 , piece_id=IT7AHX64ZIOOIFFM62EUELXD7VIHIPUQP7E4JP3UVHKGRR7B6FNQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: rpc: dial tcp 104.36.82.46:4503: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536722994 (2h22m27.317656369s) [noise=true , address=104.36.82.2:4740 , piece_id=2CK7IGYFNQN4DEGG5YNHZ6B4ECVCYZX44AEV72ZAQCLWK2F2MFOA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12wgQDbE17gP8c6kbKnNyywSkY8Hk7mchcMaymLBQzPwRsSB8QE, address:104.36.82.2:4740): protocol: rpc: dial tcp 104.36.82.2:4740: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536767955 (2h22m27.31770134s) [noise=true , address=104.36.82.47:4550 , piece_id=QFDX5TWB3USUU3G5LFV3SMQFEXXW4RKH6XUREAOOVZUYU3PPNM6A, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536812915 (2h22m27.317746296s) [noise=true , address=104.36.82.52:4104 , piece_id=XUVOXJCDWGRWDCCO2LNGIX24BROQKDOVJJD4RP4PEO26ITMGLEPA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12qmPpCmXAvuJZn2KR5Kr4nHYwKEzWvwqZ82nqNka5u1mKaZmXs, address:104.36.82.52:4104): protocol: rpc: dial tcp 104.36.82.52:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536835230 (2h22m27.317768611s) [noise=true , address=104.36.82.55:4216 , piece_id=B2QBMDBSB2CR5XL2B4M7JGG3YFUD2EWOFV5VVKBDNLKW3ON375NA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536884672 (2h22m27.317818049s) [noise=true , address=104.36.82.42:4829 , piece_id=MWASMHZGME3OJDBOFSGPDOPCSRE2R3HXAVI2UUFNYGMQPOJ22D7A, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536916240 (2h22m27.317849615s) [noise=true , address=77.90.31.18:4821 , piece_id=C33PAKSCFIWYLKC3ZMX7JSKFTEB4TID5IVKDTGHJGHHHCGL2XEYA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:44632->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536944871 (2h22m27.317878245s) [seg_pos={0 0}, upload=2170]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:14:11.559077772 (2h22m27.340011149s) [upload=2170]: Done uploading segments... Jul 27 00:14:11.559109021 (2h22m27.340042399s) [upload=2170]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.024987427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X.] duration: 592.997284ms FAIL (2025-07-27 00:14:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 593.02588ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X] duration: 30.038140627s FAIL (2025-07-27 00:14:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.038166621s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X....XX..............X.X........X...XX......................] duration: 29.376343844s FAIL (2025-07-27 00:13:56 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.376387289s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX.....X.XXX...............X..X......X.XX....X...X........] duration: 887.946534ms FAIL (2025-07-27 00:13:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 888.05571ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X.] duration: 596.519081ms FAIL (2025-07-27 00:13:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 596.559873ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X] duration: 30.025659628s FAIL (2025-07-27 00:13:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.025688641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X....XX..............X.X........X...XX.....................] duration: 23.371782548s FAIL (2025-07-27 00:13:27 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.371801055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..] duration: 1.914269926s FAIL (2025-07-27 00:13:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.914313048s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.037236912s FAIL (2025-07-27 00:13:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== CU3HOGU6TMIIKY7SNONKN2AKTKZPYT6INIAFWW7263LQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.474779599 (2h20m27.255712978s) [seg_pos={0 0}, upload=2155]: Done begin upload segment. Jul 27 00:12:11.474839535 (2h20m27.255772932s) [noise=true , address=104.36.82.47:4539 , piece_id=HPCS5VETT23V2Q2XXHJ2IV3FFHVAY3RM7E5HZ4S3CDE5FLMUZ5KQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.475380867 (2h20m27.256314255s) [noise=true , address=104.36.82.50:4042 , piece_id=IRDFI5V3RVD6LFTH6GYMTXPK6O6FGFRBX2WNHD5HUITEPTRLJXOA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.475780199 (2h20m27.256713588s) [noise=true , address=104.36.82.46:4496 , piece_id=5XADA2PAJA56CACQDFB3SR6B4MHKUU5M5BTJZCRMOJN6IX5Y2SMA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.476239578 (2h20m27.257172963s) [noise=true , address=79.127.205.245:4236 , piece_id=V6YBOCATK7SDTM3S5HAQ2MCOYH3KEROBUL63BAQGD6QW36R2EO7Q, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.476626885 (2h20m27.257560272s) [noise=true , address=79.127.220.98:4422 , piece_id=2CMSLO3DR2UCF2EZUOLXLFIVEIQ3FA52GKJJEPI2RFUK2ECSIQPQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.476998417 (2h20m27.257931803s) [noise=true , address=79.127.205.246:4273 , piece_id=TOXLEMIZKT64PLVG3R43TCTN2D5VII2ELXVPCIOGY3QPTGC2VQQA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.477370215 (2h20m27.2583036s) [noise=true , address=64.49.13.138:4483 , piece_id=4AFIYCUAOANY2OZPSUP2S4VBPV2QSJHT3B6PPL75SQOQPMAOCZ5A, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.477739767 (2h20m27.258673152s) [noise=true , address=77.90.31.13:4658 , piece_id=4MZSVUZMV6LWRTLHO6ENREQ4V2LHMZS7HSDVAJMBDIROHDNICU4Q, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.478112353 (2h20m27.259045739s) [noise=true , address=79.127.205.234:4618 , piece_id=DB62765ADIO7FHV33NTRA5OGLYOYFSA4NYL5H6YXAIBBKAMLQUQA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.478466946 (2h20m27.259400332s) [noise=true , address=104.36.82.53:4129 , piece_id=PSXK2NHZNE7JCK5EZKYZ6HW65AZWF4N6I7JAUNIIYROTSJT5G3YA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.478825049 (2h20m27.259758436s) [noise=true , address=79.127.219.38:4015 , piece_id=SO2WLO42CR43XLPEPTYN2X4DPF4X7PAWMY5WVFDTBZDGUKWHJ6IQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.479220255 (2h20m27.260153656s) [noise=true , address=104.36.82.54:4165 , piece_id=SMB6HVMSJQAI642P4HVCJJDK4EHKR3VLCERLVBVC3FTEAFRR5Q6A, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.479580033 (2h20m27.260513418s) [noise=true , address=104.36.82.13:4193 , piece_id=AHGIBTBMMANFITTACM7QMPY7ODDTVKV436YWS7AYAOQG2GJH4I5Q, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.479929192 (2h20m27.260862577s) [noise=true , address=77.90.31.21:4891 , piece_id=TRGPVOVUB7TWMWVIFLEBAI4FQH2BCPEAMKIWJ6I5HDLWFEZE6AXQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.480317609 (2h20m27.261250994s) [noise=true , address=64.49.13.138:4482 , piece_id=OWAIK3H7ATCXD3AWGCMVAN4KRML75H22M2YMWAAB7CPW4ZPDXOWA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.480670267 (2h20m27.261603657s) [noise=true , address=104.36.82.52:4116 , piece_id=KP4GODZ2LBFIOFHWWVFVCXYVY6GOXTFT7LF3NWEFBQ2O2MSSIP6A, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.487416063 (2h20m27.268349449s) [noise=true , address=77.90.31.18:4823 , piece_id=D625KEDIYDI5AKQC4IJKEIQFWHNANDDX5KNOMBTANCLLP4LNUWHA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.561344810 (2h20m27.342278187s) [upload=2155]: Next returned nil segment Jul 27 00:12:11.561363418 (2h20m27.342296794s) [upload=2155]: Waiting for error group managing segments... Jul 27 00:12:11.564909881 (2h20m27.345843249s) [noise=true , address=79.127.219.38:4015 , piece_id=SO2WLO42CR43XLPEPTYN2X4DPF4X7PAWMY5WVFDTBZDGUKWHJ6IQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.069193818 (2h20m27.850127181s) [noise=true , address=79.127.205.233:4652 , piece_id=IJUYLAWRCU3HOGU6TMIIKY7SNONKN2AKTKZPYT6INIAFWW7263LQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.119366539 (2h20m27.900299923s) [noise=true , address=79.127.205.246:4273 , piece_id=TOXLEMIZKT64PLVG3R43TCTN2D5VII2ELXVPCIOGY3QPTGC2VQQA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.121822371 (2h20m27.902755735s) [noise=true , address=79.127.205.245:4236 , piece_id=V6YBOCATK7SDTM3S5HAQ2MCOYH3KEROBUL63BAQGD6QW36R2EO7Q, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.140666801 (2h20m27.921600169s) [noise=true , address=79.127.205.234:4618 , piece_id=DB62765ADIO7FHV33NTRA5OGLYOYFSA4NYL5H6YXAIBBKAMLQUQA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.715547694 (2h20m28.49648108s) [noise=true , address=77.90.31.13:4658 , piece_id=4MZSVUZMV6LWRTLHO6ENREQ4V2LHMZS7HSDVAJMBDIROHDNICU4Q, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:14.641219316 (2h20m30.4221527s) [noise=true , address=79.127.220.98:4422 , piece_id=2CMSLO3DR2UCF2EZUOLXLFIVEIQ3FA52GKJJEPI2RFUK2ECSIQPQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:21.456468361 (2h20m37.237401727s) [noise=true , address=64.49.13.138:4483 , piece_id=4AFIYCUAOANY2OZPSUP2S4VBPV2QSJHT3B6PPL75SQOQPMAOCZ5A, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:22.866703724 (2h20m38.647637086s) [noise=true , address=64.49.13.138:4482 , piece_id=OWAIK3H7ATCXD3AWGCMVAN4KRML75H22M2YMWAAB7CPW4ZPDXOWA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:13:11.396553355 (2h21m27.177486746s) [noise=true , address=104.36.82.13:4193 , piece_id=AHGIBTBMMANFITTACM7QMPY7ODDTVKV436YWS7AYAOQG2GJH4I5Q, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:15XNLgptmXprcfkVMJsuzY2WDyHiZDpLetiBpmjXhb4c9q7yMd, address:104.36.82.13:4193): protocol: rpc: dial tcp 104.36.82.13:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.396683683 (2h21m27.177617072s) [noise=true , address=104.36.82.53:4129 , piece_id=PSXK2NHZNE7JCK5EZKYZ6HW65AZWF4N6I7JAUNIIYROTSJT5G3YA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.396752474 (2h21m27.177685853s) [noise=true , address=104.36.82.54:4165 , piece_id=SMB6HVMSJQAI642P4HVCJJDK4EHKR3VLCERLVBVC3FTEAFRR5Q6A, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.396819753 (2h21m27.177753133s) [noise=true , address=77.90.31.21:4891 , piece_id=TRGPVOVUB7TWMWVIFLEBAI4FQH2BCPEAMKIWJ6I5HDLWFEZE6AXQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.397226423 (2h21m27.178159809s) [noise=true , address=77.90.31.18:4823 , piece_id=D625KEDIYDI5AKQC4IJKEIQFWHNANDDX5KNOMBTANCLLP4LNUWHA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:13:11.397353794 (2h21m27.178287173s) [noise=true , address=104.36.82.46:4496 , piece_id=5XADA2PAJA56CACQDFB3SR6B4MHKUU5M5BTJZCRMOJN6IX5Y2SMA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:13:11.397444729 (2h21m27.178378109s) [noise=true , address=104.36.82.47:4539 , piece_id=HPCS5VETT23V2Q2XXHJ2IV3FFHVAY3RM7E5HZ4S3CDE5FLMUZ5KQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:13:11.397919985 (2h21m27.178853365s) [noise=true , address=104.36.82.50:4042 , piece_id=IRDFI5V3RVD6LFTH6GYMTXPK6O6FGFRBX2WNHD5HUITEPTRLJXOA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:1LbRrZsqp1TR354jZQLstDmWAUBVBMyid8AuAbTfQj7N89Mnte, address:104.36.82.50:4042): protocol: rpc: dial tcp 104.36.82.50:4042: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.397990342 (2h21m27.17892372s) [noise=true , address=104.36.82.52:4116 , piece_id=KP4GODZ2LBFIOFHWWVFVCXYVY6GOXTFT7LF3NWEFBQ2O2MSSIP6A, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12vKCXgWTAsipTvUK9XA3sL6DznFA8BqZc6WCN84PBuLhKCTT9m, address:104.36.82.52:4116): protocol: write tcp 159.223.143.39:37602->104.36.82.52:4116: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.398015537 (2h21m27.178948918s) [seg_pos={0 0}, upload=2155]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:13:11.429641576 (2h21m27.210574947s) [upload=2155]: Done uploading segments... Jul 27 00:13:11.429669188 (2h21m27.210602576s) [upload=2155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.037262861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X....XX..............X.X........X...XX....................] duration: 46.84525729s FAIL (2025-07-27 00:13:04 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.845300214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XX.....X.XXX...............X..X......X.XX....X...X.......] duration: 551.263327ms FAIL (2025-07-27 00:12:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.285881ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.] duration: 1.935421076s FAIL (2025-07-27 00:12:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 314542506 (2h19m48.095475886s) [upload=2145]: Done starting upload Jul 27 00:11:32.314833287 (2h19m48.095766663s) [upload=2145]: Uploading segments... Jul 27 00:11:32.314893632 (2h19m48.095827017s) [upload=2145]: Got next segment. Inline: false Jul 27 00:11:32.314934607 (2h19m48.095867986s) [upload=2145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:11:32.314940536 (2h19m48.095873952s) [upload=2145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:11:32.358989266 (2h19m48.139922631s) [seg_pos={0 0}, upload=2145]: Begin upload segment... Jul 27 00:11:32.359080795 (2h19m48.14001416s) [seg_pos={0 0}, upload=2145]: Done begin upload segment. Jul 27 00:11:32.359112107 (2h19m48.140045483s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359131009 (2h19m48.140064388s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359557235 (2h19m48.14049062s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359672407 (2h19m48.140605785s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359868527 (2h19m48.140801904s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360315728 (2h19m48.141249109s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360940298 (2h19m48.141873678s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363171520 (2h19m48.144104901s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363609540 (2h19m48.144542915s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363934473 (2h19m48.144867872s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364418032 (2h19m48.145351412s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364847636 (2h19m48.145781026s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365229070 (2h19m48.146162462s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365580258 (2h19m48.146513638s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365888285 (2h19m48.146821663s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.366216950 (2h19m48.147150328s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.435644764 (2h19m48.216578142s) [upload=2145]: Next returned nil segment Jul 27 00:11:32.435666271 (2h19m48.216599637s) [upload=2145]: Waiting for error group managing segments... Jul 27 00:11:32.440416261 (2h19m48.221349639s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440451732 (2h19m48.221385102s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440950422 (2h19m48.221883793s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440987609 (2h19m48.221920979s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.848839884 (2h19m48.629773253s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.836650446 (2h19m49.617583807s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.837905779 (2h19m49.618839149s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:12:02.285759366 (2h20m18.066692752s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285860565 (2h20m18.066793953s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285936009 (2h20m18.066869378s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285991545 (2h20m18.066924913s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286048764 (2h20m18.066982132s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286105406 (2h20m18.067038773s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286160478 (2h20m18.067093844s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286228850 (2h20m18.067162216s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286292018 (2h20m18.067225381s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 159.223.143.39:34508->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286313912 (2h20m18.067247312s) [seg_pos={0 0}, upload=2145]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:12:02.311015245 (2h20m18.091948638s) [upload=2145]: Done uploading segments... Jul 27 00:12:02.311076840 (2h20m18.092010226s) [upload=2145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 1.935456927s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...XX.....X.XXX...............X..X......X.XX....X...X......] duration: 29.59103493s FAIL (2025-07-27 00:12:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.591080299s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)