Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 300164 reports (Page 5505 of 6004)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........X.X........X...XX...............................X..] duration: 37.270035113s FAIL (2025-07-27 00:24:35 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.270062145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033393171s FAIL (2025-07-27 00:24:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 56MCVOLGTGD5SI7H47F7ISMKWQ, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.068393908 (2h31m41.849327276s) [noise=true , address=77.90.31.16:4754 , piece_id=OTMENKECVDUSRPI4E6XOKUR6I5RLFXARL6GMWGLRVADL3UEWO3RA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.068435197 (2h31m41.849368577s) [noise=true , address=79.127.219.38:4004 , piece_id=A3KLNDYIXZYJSHXDDBM6KDTJA3WBY2PQQVAG3AHUFYUGKMKJWH6Q, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.068934312 (2h31m41.849867697s) [noise=true , address=104.36.82.15:4281 , piece_id=HOLAPLJHZ7ETGQBXXJEH6XJ6DNMIKLO725CVZBMJSCHTNEM2B4SQ, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.069392366 (2h31m41.850325747s) [noise=true , address=79.127.205.234:4618 , piece_id=TNKJNH7KY7VSDZH7F4CX364TY3USEH3X6VOF6ZQNVDZWYEKRJFLQ, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.069716819 (2h31m41.850650187s) [noise=true , address=104.36.82.56:4242 , piece_id=H7SMMJXQDU6XSH4KOJ22CIRNGV6WOMILRSXA4RNTKF46EUMR3B3A, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.069889355 (2h31m41.850822841s) [noise=true , address=77.90.31.15:4727 , piece_id=GEBZJ2R7EBBSUP4KVKAIGPSKTCSODGKMJEVCHBIJSJESPMFYI7MQ, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.070316539 (2h31m41.851249923s) [noise=true , address=104.36.82.44:4888 , piece_id=B62KC75RXRBQ5Q4OS4RLXOSHITLMMTJST52KUV3PRCKBJ33GPFOQ, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.070660719 (2h31m41.851594097s) [noise=true , address=104.36.82.46:4510 , piece_id=TMDPJEIAYODA27N4XYCY6PHLYA5K27HSTA54YD22CPVKT3NONX3A, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.071188191 (2h31m41.852121574s) [noise=true , address=79.127.201.215:4351 , piece_id=Z263YN7U7VBSO6L7NH7K5ALDNUN27BTZDKRROU6PP45E37IGNIUA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.071581001 (2h31m41.85251438s) [noise=true , address=64.49.13.138:4482 , piece_id=5A3J7FOOV22OBELPOWNH7UJBB5VSDSR45FUBG6ZULZ3P5S7K3AYQ, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.072037758 (2h31m41.852971144s) [noise=true , address=79.127.205.245:4239 , piece_id=AKDBKKB5ISBHKP4BIEOOR4CO6OLQ3SBBRHL43ZULPRQG2LZ23ASA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.072186947 (2h31m41.853120331s) [noise=true , address=77.90.31.10:4540 , piece_id=ZH2L76FPP24Y36STCI6AJW4JVNABCM4MRW7ENX27JH53XP2RZXIA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.072608764 (2h31m41.853542147s) [noise=true , address=104.36.82.50:4048 , piece_id=GQUJDQ4EHLBKBA7GTC7KIVQ3A3YE3SKA5POJL543GMA37OETE7ZA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.073047059 (2h31m41.853980442s) [noise=true , address=79.127.201.214:4321 , piece_id=HWDULUJQG5ZKA5WLVVR2RX6DOMR7SLLUZGQKHO7UTLFL2MTONQRA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.073644658 (2h31m41.854578032s) [noise=true , address=104.36.82.47:4517 , piece_id=X3RVSICOB7DRHEMMRUG6Q7GG3XR5DKZYHMBAXRHHCDY3NWJUTWRA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.074244739 (2h31m41.855178123s) [noise=true , address=104.36.82.54:4171 , piece_id=HA4FPTMKKADZFA4LHBKJWIWVEGRINAQ253M2JIO3K4QGNNJTGEWA, seg_pos={0 0}, upload=2331]: Uploading piece... Jul 27 00:23:26.147711829 (2h31m41.928645211s) [upload=2331]: Next returned nil segment Jul 27 00:23:26.148916453 (2h31m41.929849843s) [upload=2331]: Waiting for error group managing segments... Jul 27 00:23:26.197757268 (2h31m41.978690638s) [noise=true , address=77.90.31.15:4727 , piece_id=GEBZJ2R7EBBSUP4KVKAIGPSKTCSODGKMJEVCHBIJSJESPMFYI7MQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:23:26.275108732 (2h31m42.05604211s) [noise=true , address=79.127.219.38:4004 , piece_id=A3KLNDYIXZYJSHXDDBM6KDTJA3WBY2PQQVAG3AHUFYUGKMKJWH6Q, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:23:26.653390226 (2h31m42.434323603s) [noise=true , address=79.127.205.246:4266 , piece_id=DE7LHDVHS34DSHZA6RYUQ3PVP656MCVOLGTGD5SI7H47F7ISMKWQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:23:26.664941515 (2h31m42.445874883s) [noise=true , address=79.127.205.245:4239 , piece_id=AKDBKKB5ISBHKP4BIEOOR4CO6OLQ3SBBRHL43ZULPRQG2LZ23ASA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:23:26.701941806 (2h31m42.482875179s) [noise=true , address=79.127.205.234:4618 , piece_id=TNKJNH7KY7VSDZH7F4CX364TY3USEH3X6VOF6ZQNVDZWYEKRJFLQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:23:30.267346655 (2h31m46.048280039s) [noise=true , address=79.127.201.214:4321 , piece_id=HWDULUJQG5ZKA5WLVVR2RX6DOMR7SLLUZGQKHO7UTLFL2MTONQRA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:23:32.394608110 (2h31m48.1755415s) [noise=true , address=64.49.13.138:4482 , piece_id=5A3J7FOOV22OBELPOWNH7UJBB5VSDSR45FUBG6ZULZ3P5S7K3AYQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:23:38.279720172 (2h31m54.060653559s) [noise=true , address=45.91.116.14:4206 , piece_id=VS7D3DL6CV6WZ5UKEBW4ZMIPRNHTDVYTEMVL6BL7HSQ2KAECG5EA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:24:01.540097748 (2h32m17.321031132s) [noise=true , address=79.127.201.215:4351 , piece_id=Z263YN7U7VBSO6L7NH7K5ALDNUN27BTZDKRROU6PP45E37IGNIUA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil> Jul 27 00:24:25.973451469 (2h32m41.754384845s) [noise=true , address=104.36.82.15:4281 , piece_id=HOLAPLJHZ7ETGQBXXJEH6XJ6DNMIKLO725CVZBMJSCHTNEM2B4SQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2, address:104.36.82.15:4281): protocol: rpc: dial tcp 104.36.82.15:4281: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.973519077 (2h32m41.754452454s) [noise=true , address=77.90.31.10:4540 , piece_id=ZH2L76FPP24Y36STCI6AJW4JVNABCM4MRW7ENX27JH53XP2RZXIA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.973593522 (2h32m41.754526899s) [noise=true , address=104.36.82.50:4048 , piece_id=GQUJDQ4EHLBKBA7GTC7KIVQ3A3YE3SKA5POJL543GMA37OETE7ZA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1rJd5FahWjKfvJstcoLE5uJGgtVj2uZirPpCJxbfGPGQ3j4qxC, address:104.36.82.50:4048): protocol: write tcp 159.223.143.39:60142->104.36.82.50:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.973656555 (2h32m41.754589931s) [noise=true , address=104.36.82.56:4242 , piece_id=H7SMMJXQDU6XSH4KOJ22CIRNGV6WOMILRSXA4RNTKF46EUMR3B3A, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: rpc: dial tcp 104.36.82.56:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.973849672 (2h32m41.754783048s) [noise=true , address=104.36.82.44:4888 , piece_id=B62KC75RXRBQ5Q4OS4RLXOSHITLMMTJST52KUV3PRCKBJ33GPFOQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: write tcp 159.223.143.39:50010->104.36.82.44:4888: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.973909573 (2h32m41.754842944s) [noise=true , address=104.36.82.54:4171 , piece_id=HA4FPTMKKADZFA4LHBKJWIWVEGRINAQ253M2JIO3K4QGNNJTGEWA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: write tcp 159.223.143.39:55640->104.36.82.54:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.973940294 (2h32m41.754873673s) [noise=true , address=77.90.31.16:4754 , piece_id=OTMENKECVDUSRPI4E6XOKUR6I5RLFXARL6GMWGLRVADL3UEWO3RA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.974020126 (2h32m41.754953512s) [noise=true , address=104.36.82.46:4510 , piece_id=TMDPJEIAYODA27N4XYCY6PHLYA5K27HSTA54YD22CPVKT3NONX3A, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: write tcp 159.223.143.39:42322->104.36.82.46:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.974069289 (2h32m41.755002661s) [noise=true , address=104.36.82.47:4517 , piece_id=X3RVSICOB7DRHEMMRUG6Q7GG3XR5DKZYHMBAXRHHCDY3NWJUTWRA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: write tcp 159.223.143.39:49678->104.36.82.47:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:25.974111876 (2h32m41.755045248s) [seg_pos={0 0}, upload=2331]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:24:26.001491819 (2h32m41.782425201s) [upload=2331]: Done uploading segments... Jul 27 00:24:26.001524618 (2h32m41.782457986s) [upload=2331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.033431376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...............X..X......X.XX....X...X.................XX..] duration: 1.839939307s FAIL (2025-07-27 00:24:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.839971886s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.] duration: 2.497043876s FAIL (2025-07-27 00:23:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.497143537s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX] duration: 30.028014755s FAIL (2025-07-27 00:23:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.028043855s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXX] duration: 30.032367125s FAIL (2025-07-27 00:23:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== load enough pieces (needed at least 13 but got 10) Jul 27 00:23:05.830875957 (2h31m21.611809333s) [upload=2325]: Starting upload Jul 27 00:23:05.830976568 (2h31m21.611909942s) [upload=2325]: Done starting upload Jul 27 00:23:05.831142029 (2h31m21.612075403s) [upload=2325]: Uploading segments... Jul 27 00:23:05.831520163 (2h31m21.61245353s) [upload=2325]: Got next segment. Inline: false Jul 27 00:23:05.831589712 (2h31m21.612523088s) [upload=2325]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:23:05.831595331 (2h31m21.612528701s) [upload=2325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:23:05.859493989 (2h31m21.640427375s) [seg_pos={0 0}, upload=2325]: Begin upload segment... Jul 27 00:23:05.859548834 (2h31m21.640482209s) [seg_pos={0 0}, upload=2325]: Done begin upload segment. Jul 27 00:23:05.859597857 (2h31m21.640531241s) [noise=true , address=45.91.116.7:4009 , piece_id=PC4BDAPBLCJO7UJEQMQUV23TYFPUEF3GCO5KGBUWBAF5OULRYWWA, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.859628700 (2h31m21.640562082s) [noise=true , address=104.36.82.6:4875 , piece_id=R3L46YYQCH74UYBV6FJUENTZXSBEO75W7NUZVB2IEUEXSCHNSZWQ, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.859697244 (2h31m21.640630612s) [noise=true , address=45.91.116.10:4083 , piece_id=JHWXXNPRVODBPGIYGVSWTHFKK43FTUJKUMUYFEDJO4LVCUW3ATDQ, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.860273748 (2h31m21.641207133s) [noise=true , address=77.90.31.13:4627 , piece_id=GBFZPOPW44JHI7GXWBVMT7T7U7GJ7ZQY5BMLBC4U5NFT36VJ3S6A, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.860328404 (2h31m21.641261772s) [noise=true , address=45.91.116.14:4189 , piece_id=HECENZE2ESPYKNG32TMADTVMBBKPBAYASHSSWC4EUJK3K3U3MP6A, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.860841565 (2h31m21.64177496s) [noise=true , address=104.36.82.6:4874 , piece_id=GSERE7IZD3WUOAL4BBW2GWKNMMQWRM3HHYBDXRALEEJLAIQEZ6XA, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.860853636 (2h31m21.641787023s) [noise=true , address=77.90.31.18:4805 , piece_id=4FLYNQHNAEIVPRPXFUJPSOFEJUMRHH6LSCOIVR7AZSZ4W7KAZZEQ, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.861242275 (2h31m21.642175655s) [noise=true , address=77.90.31.17:4771 , piece_id=BLV6ZGDEBKCWTELYBFORKGF2A2IMW5ILBB6CF2S2QHBOTALKW4BQ, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.861317433 (2h31m21.642250803s) [noise=true , address=104.36.82.5:4838 , piece_id=5RLMP5ZARSNRBOKXU5RYA65FW6BRKKWO7JVZGKKOH3NYWOOCFQRA, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.861624060 (2h31m21.642557447s) [noise=true , address=77.90.31.10:4518 , piece_id=JNOXAEXTO2XR5R262KIHPQKXTX4J5LWBYVCHMY4RJINLOOA775TQ, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.861970118 (2h31m21.642903501s) [noise=true , address=104.36.82.10:4077 , piece_id=L6TYU6DTHHHBM3AGOXE5IY5JFI2CQ53UYIR7MZMBGWU3N5G2AVVQ, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.862371652 (2h31m21.643305032s) [noise=true , address=77.90.31.19:4841 , piece_id=Q2TOOY7WLFPLAUS4SF3Y2VARO6FKDLM5KOG4N3GOZIPQVCO43DPQ, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.863110035 (2h31m21.644043404s) [noise=true , address=104.36.82.12:4149 , piece_id=SSDAXUUCREBIUHE7SJ3UPMLPL45VUBFUVPUIBFYXTQEK553F4AAA, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.863579173 (2h31m21.644512549s) [noise=true , address=45.91.116.9:4262 , piece_id=DHLZBDPP5N23JLGLGB4S4M57XU2WHEDUW4NIK33J2YXSTQBFCO5Q, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.863673501 (2h31m21.644606871s) [noise=true , address=104.36.82.7:4911 , piece_id=ABESGXHYNDJ7OL27XBRDVNB3MCPOF3CFVZV6X2A7KRBTNGWCOY3Q, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.864361977 (2h31m21.645295363s) [noise=true , address=77.90.31.14:4663 , piece_id=6DY3BNYAGEHJTTREC7OLQ2AF37MKL7IVYUV7ERK4ZRI73MVTWU7Q, seg_pos={0 0}, upload=2325]: Uploading piece... Jul 27 00:23:05.944508944 (2h31m21.725442325s) [upload=2325]: Next returned nil segment Jul 27 00:23:05.944531926 (2h31m21.72546531s) [upload=2325]: Waiting for error group managing segments... Jul 27 00:23:05.949015550 (2h31m21.729948926s) [noise=true , address=45.91.116.7:4009 , piece_id=PC4BDAPBLCJO7UJEQMQUV23TYFPUEF3GCO5KGBUWBAF5OULRYWWA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:05.952473241 (2h31m21.733406612s) [noise=true , address=45.91.116.10:4083 , piece_id=JHWXXNPRVODBPGIYGVSWTHFKK43FTUJKUMUYFEDJO4LVCUW3ATDQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:05.952908571 (2h31m21.73384194s) [noise=true , address=77.90.31.14:4663 , piece_id=6DY3BNYAGEHJTTREC7OLQ2AF37MKL7IVYUV7ERK4ZRI73MVTWU7Q, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:05.953379182 (2h31m21.734312558s) [noise=true , address=45.91.116.14:4189 , piece_id=HECENZE2ESPYKNG32TMADTVMBBKPBAYASHSSWC4EUJK3K3U3MP6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:05.953872699 (2h31m21.734806069s) [noise=true , address=45.91.116.9:4262 , piece_id=DHLZBDPP5N23JLGLGB4S4M57XU2WHEDUW4NIK33J2YXSTQBFCO5Q, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:05.954263594 (2h31m21.735196968s) [noise=true , address=77.90.31.18:4805 , piece_id=4FLYNQHNAEIVPRPXFUJPSOFEJUMRHH6LSCOIVR7AZSZ4W7KAZZEQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:07.351282983 (2h31m23.132216363s) [noise=true , address=77.90.31.19:4841 , piece_id=Q2TOOY7WLFPLAUS4SF3Y2VARO6FKDLM5KOG4N3GOZIPQVCO43DPQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:07.370174561 (2h31m23.151107923s) [noise=true , address=77.90.31.13:4627 , piece_id=GBFZPOPW44JHI7GXWBVMT7T7U7GJ7ZQY5BMLBC4U5NFT36VJ3S6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil> Jul 27 00:23:35.801690340 (2h31m51.582623727s) [noise=true , address=104.36.82.10:4077 , piece_id=L6TYU6DTHHHBM3AGOXE5IY5JFI2CQ53UYIR7MZMBGWU3N5G2AVVQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.801789541 (2h31m51.582722919s) [noise=true , address=77.90.31.17:4771 , piece_id=BLV6ZGDEBKCWTELYBFORKGF2A2IMW5ILBB6CF2S2QHBOTALKW4BQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.801851272 (2h31m51.58278465s) [noise=true , address=104.36.82.7:4911 , piece_id=ABESGXHYNDJ7OL27XBRDVNB3MCPOF3CFVZV6X2A7KRBTNGWCOY3Q, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:41292->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.801921382 (2h31m51.582854761s) [noise=true , address=104.36.82.12:4149 , piece_id=SSDAXUUCREBIUHE7SJ3UPMLPL45VUBFUVPUIBFYXTQEK553F4AAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.801960525 (2h31m51.582893887s) [noise=true , address=104.36.82.5:4838 , piece_id=5RLMP5ZARSNRBOKXU5RYA65FW6BRKKWO7JVZGKKOH3NYWOOCFQRA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:59918->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.801983053 (2h31m51.58291643s) [noise=true , address=104.36.82.6:4874 , piece_id=GSERE7IZD3WUOAL4BBW2GWKNMMQWRM3HHYBDXRALEEJLAIQEZ6XA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.802023711 (2h31m51.582957089s) [noise=true , address=77.90.31.10:4518 , piece_id=JNOXAEXTO2XR5R262KIHPQKXTX4J5LWBYVCHMY4RJINLOOA775TQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.802048687 (2h31m51.582982065s) [noise=true , address=104.36.82.6:4875 , piece_id=R3L46YYQCH74UYBV6FJUENTZXSBEO75W7NUZVB2IEUEXSCHNSZWQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:53440->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.802082296 (2h31m51.583015681s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:23:35.830966119 (2h31m51.611899487s) [upload=2325]: Done uploading segments... Jul 27 00:23:35.830993756 (2h31m51.611927137s) [upload=2325]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.032387822s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX...............X..X......X.XX....X...X.................XX.] duration: 9.561507357s FAIL (2025-07-27 00:23:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.561556497s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031647854s FAIL (2025-07-27 00:23:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== item: pb.BatchRequestItem_ObjectBegin Jul 27 00:22:25.898161176 (2h30m41.679094553s) [upload=2313]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:22:25.940641206 (2h30m41.721574593s) [seg_pos={0 0}, upload=2313]: Begin upload segment... Jul 27 00:22:25.940716081 (2h30m41.721649443s) [seg_pos={0 0}, upload=2313]: Done begin upload segment. Jul 27 00:22:25.940767132 (2h30m41.721700518s) [noise=true , address=104.36.82.19:4431 , piece_id=3HZC5ON4OKUMVROACB4CSFHB6B64Z6CH7YSYSNKZ7JLH3V7X4JWA, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.941254472 (2h30m41.722187859s) [noise=true , address=104.36.82.52:4096 , piece_id=RB3SRXV6TURCZC24MNNKYNRNHAK7HTZ5IVQKAU5T432CGETFFOLQ, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.941669697 (2h30m41.722603077s) [noise=true , address=77.90.31.21:4907 , piece_id=VGN4MO5V6AGFAFHXLPMSGAC23KET27OR3SMX54NVVC523ZBAKUNQ, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.942027107 (2h30m41.722960497s) [noise=true , address=64.49.13.137:4502 , piece_id=REKSKHRQUS6VGKYDEJKC73NRZCY6GVNOGMY23WW5KO6SWBTWZL4Q, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.942390382 (2h30m41.723323763s) [noise=true , address=79.127.220.101:4048 , piece_id=CFAHURCN6KJ4JW4HXQNI74BQ3GIJUOJWZDVBKEV5HD4H6OAIMX5A, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.942698295 (2h30m41.723631674s) [noise=true , address=64.49.13.140:4508 , piece_id=G6QOJIAJEIOUZGE23GPOW23ESGM5AZXBEGVYC2BKD3JAII52RZJA, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.942981866 (2h30m41.723915246s) [noise=true , address=64.49.13.26:4953 , piece_id=UVEUWX2HWLBAPGKMPVTZXDWE46655QSOSWJYNBATEAR2DNCNTHCQ, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.943384935 (2h30m41.724318317s) [noise=true , address=64.49.13.139:4478 , piece_id=QCUOTMGT547UX5OFD5J3DGPV4II4VMTX46UEVIJQXLM4XVDHB7NA, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.943582481 (2h30m41.724515884s) [noise=true , address=45.91.116.9:4277 , piece_id=VFV26NRL4V63FM6OVVO7YSV2HEKVX3IMNLYVADFGQ53PVX5X5B2Q, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.943729340 (2h30m41.724662729s) [noise=true , address=79.127.219.38:4018 , piece_id=DWI35RTF2QE365DMZUU7WFEJRUHIRZEGQEOHZBBQNLIJ3ZFKQSSQ, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.944218928 (2h30m41.725152351s) [noise=true , address=79.127.205.235:4691 , piece_id=NHXU5I6RHDFMMQD5FIF2VBPA6Q2KID6LOIYJPTJTQALAOLJ7DDTA, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.944620429 (2h30m41.725553816s) [noise=true , address=79.127.205.233:4646 , piece_id=FB3DEPJHM3A7USMLM22MPDMJYZIY4BSQXXEZNPNWJF4LLCCCL7MQ, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.945039019 (2h30m41.725972416s) [noise=true , address=104.36.82.46:4501 , piece_id=62HAXOPLI344F32UV2ZRGEOQQW5LAKF6W3RKRKEKU6YBGY6R2POA, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.945435137 (2h30m41.726368527s) [noise=true , address=104.36.82.56:4258 , piece_id=ABG5XOANCNDLJ6TICN7NRRZTXXGYBINDKXKN64KXNCY3BIV56Z3Q, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.945798530 (2h30m41.72673192s) [noise=true , address=64.49.13.138:4483 , piece_id=X52L3JLMDKSHGDMV2KKZBFKAC7NYYHZSEHPGMFXKKAUPHX3SLVPQ, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.946174758 (2h30m41.727108146s) [noise=true , address=45.91.116.12:4231 , piece_id=QV7RSQIL3KQWGBLYYOEVZU55FBIYPYPQTQ2XHOGLVMPWRPTYGZ5A, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.946547191 (2h30m41.727480579s) [noise=true , address=77.90.31.15:4722 , piece_id=GLXCKILWSMI37DTREARWYCST7PUBYJEDT3CCJGNMYDY6XKR3JTRQ, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:25.951701239 (2h30m41.732634631s) [noise=true , address=79.127.205.245:4224 , piece_id=YCYZL46TMOVBGYSY2HRPGHERNWBDLJWJMRSWUTJLOIQN6ODM7YHA, seg_pos={0 0}, upload=2313]: Uploading piece... Jul 27 00:22:26.067329858 (2h30m41.848263222s) [upload=2313]: Next returned nil segment Jul 27 00:22:26.067353897 (2h30m41.848287264s) [upload=2313]: Waiting for error group managing segments... Jul 27 00:22:26.069021324 (2h30m41.849954711s) [noise=true , address=79.127.220.101:4048 , piece_id=CFAHURCN6KJ4JW4HXQNI74BQ3GIJUOJWZDVBKEV5HD4H6OAIMX5A, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:22:26.074424607 (2h30m41.855357969s) [noise=true , address=79.127.219.38:4018 , piece_id=DWI35RTF2QE365DMZUU7WFEJRUHIRZEGQEOHZBBQNLIJ3ZFKQSSQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:26.109637102 (2h30m41.89057048s) [noise=true , address=77.90.31.15:4722 , piece_id=GLXCKILWSMI37DTREARWYCST7PUBYJEDT3CCJGNMYDY6XKR3JTRQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:26.568493158 (2h30m42.349426536s) [noise=true , address=79.127.205.235:4691 , piece_id=NHXU5I6RHDFMMQD5FIF2VBPA6Q2KID6LOIYJPTJTQALAOLJ7DDTA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:26.617213670 (2h30m42.398147053s) [noise=true , address=79.127.205.233:4646 , piece_id=FB3DEPJHM3A7USMLM22MPDMJYZIY4BSQXXEZNPNWJF4LLCCCL7MQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:26.629366290 (2h30m42.410299667s) [noise=true , address=79.127.205.245:4224 , piece_id=YCYZL46TMOVBGYSY2HRPGHERNWBDLJWJMRSWUTJLOIQN6ODM7YHA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:26.642448415 (2h30m42.423381805s) [noise=true , address=64.49.13.137:4502 , piece_id=REKSKHRQUS6VGKYDEJKC73NRZCY6GVNOGMY23WW5KO6SWBTWZL4Q, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:27.151769117 (2h30m42.9327025s) [noise=true , address=45.91.116.9:4277 , piece_id=VFV26NRL4V63FM6OVVO7YSV2HEKVX3IMNLYVADFGQ53PVX5X5B2Q, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:22:28.290134394 (2h30m44.071067776s) [noise=true , address=64.49.13.140:4508 , piece_id=G6QOJIAJEIOUZGE23GPOW23ESGM5AZXBEGVYC2BKD3JAII52RZJA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:29.540568553 (2h30m45.321501939s) [noise=true , address=45.91.116.12:4231 , piece_id=QV7RSQIL3KQWGBLYYOEVZU55FBIYPYPQTQ2XHOGLVMPWRPTYGZ5A, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:31.674930017 (2h30m47.455863397s) [noise=true , address=64.49.13.26:4953 , piece_id=UVEUWX2HWLBAPGKMPVTZXDWE46655QSOSWJYNBATEAR2DNCNTHCQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:31.791830817 (2h30m47.572764186s) [noise=true , address=64.49.13.138:4483 , piece_id=X52L3JLMDKSHGDMV2KKZBFKAC7NYYHZSEHPGMFXKKAUPHX3SLVPQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:22:33.069396553 (2h30m48.850329917s) [noise=true , address=64.49.13.139:4478 , piece_id=QCUOTMGT547UX5OFD5J3DGPV4II4VMTX46UEVIJQXLM4XVDHB7NA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil> Jul 27 00:23:25.867516117 (2h31m41.648449487s) [noise=true , address=104.36.82.56:4258 , piece_id=ABG5XOANCNDLJ6TICN7NRRZTXXGYBINDKXKN64KXNCY3BIV56Z3Q, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: rpc: dial tcp 104.36.82.56:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.867575536 (2h31m41.648508902s) [noise=true , address=104.36.82.46:4501 , piece_id=62HAXOPLI344F32UV2ZRGEOQQW5LAKF6W3RKRKEKU6YBGY6R2POA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 159.223.143.39:47734->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.867621495 (2h31m41.648554865s) [noise=true , address=77.90.31.21:4907 , piece_id=VGN4MO5V6AGFAFHXLPMSGAC23KET27OR3SMX54NVVC523ZBAKUNQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: rpc: dial tcp 77.90.31.21:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.867711307 (2h31m41.648644687s) [noise=true , address=104.36.82.52:4096 , piece_id=RB3SRXV6TURCZC24MNNKYNRNHAK7HTZ5IVQKAU5T432CGETFFOLQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.867803587 (2h31m41.64873695s) [noise=true , address=104.36.82.19:4431 , piece_id=3HZC5ON4OKUMVROACB4CSFHB6B64Z6CH7YSYSNKZ7JLH3V7X4JWA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 159.223.143.39:39806->104.36.82.19:4431: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.867837251 (2h31m41.648770639s) [seg_pos={0 0}, upload=2313]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:23:25.896823463 (2h31m41.677756839s) [upload=2313]: Done uploading segments... Jul 27 00:23:25.896860064 (2h31m41.677793443s) [upload=2313]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.031668044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........X.X........X...XX...............................X.] duration: 28.582429707s FAIL (2025-07-27 00:23:25 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.582454163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX...............X..X......X.XX....X...X.................XX] duration: 30.001531855s FAIL (2025-07-27 00:23:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00154849s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXX] duration: 30.026156574s FAIL (2025-07-27 00:23:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== WE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:58496->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:06.992951239 (2h30m22.773884626s) [seg_pos={0 0}, upload=2302]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:22:07.017432150 (2h30m22.798365536s) [upload=2302]: Done uploading segments... Jul 27 00:22:07.017466365 (2h30m22.79839975s) [upload=2302]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:22:35.735772116 (2h30m51.516705485s) [upload=2316]: Starting upload Jul 27 00:22:35.735889547 (2h30m51.516822913s) [upload=2316]: Done starting upload Jul 27 00:22:35.736136946 (2h30m51.517070315s) [upload=2316]: Uploading segments... Jul 27 00:22:35.736579374 (2h30m51.517512741s) [upload=2316]: Got next segment. Inline: false Jul 27 00:22:35.736649272 (2h30m51.517582638s) [upload=2316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:22:35.736656057 (2h30m51.517589425s) [upload=2316]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:22:35.773528464 (2h30m51.554461842s) [seg_pos={0 0}, upload=2316]: Begin upload segment... Jul 27 00:22:35.773571359 (2h30m51.554504724s) [seg_pos={0 0}, upload=2316]: Done begin upload segment. Jul 27 00:22:35.773607061 (2h30m51.554540439s) [noise=true , address=104.36.82.2:4731 , piece_id=TAHBPPGVULUOM3KNMPUWBGLHUJMETCHHBPWZSP5VRFG3GSTEZSGA, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.774052025 (2h30m51.554985399s) [noise=true , address=77.90.31.21:4878 , piece_id=27RVSTANPTACCVI7UO2LIM2K46SIGQX4A47X7AHKZP2KQPK5XHUA, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.774691559 (2h30m51.555624941s) [noise=true , address=45.91.116.11:4119 , piece_id=V4BNG3X6KOITLDNCFWT6XBDZOJCYDXO5H3N6AJ7MKKF4NXMP72AQ, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.776464301 (2h30m51.557397688s) [noise=true , address=77.90.31.10:4517 , piece_id=4TEO72K4T5IBB3KAYDJ7XA3BMPSOIX4FGLOSM3AYBDVSKRVPGX7A, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.776835877 (2h30m51.55776926s) [noise=true , address=104.36.82.16:4295 , piece_id=3CNUVGW4XWZB4PMYKNWVKHKNRQH5M4XYWLULZ6HPY4GYBJBUYWIQ, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.777053841 (2h30m51.557987254s) [noise=true , address=77.90.31.14:4663 , piece_id=S36NSRGS7IGRKBIOKHLZGMZLIKMXZRSCVZS5WJZIIP7FLGONKLHA, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.777214768 (2h30m51.558148154s) [noise=true , address=104.36.82.14:4223 , piece_id=364455LQ6FYYIA6II4JS7PKFZM7422U2B2KD4DD3VGYVJPQ2YZWA, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.777525863 (2h30m51.558459249s) [noise=true , address=45.91.116.10:4082 , piece_id=OA25WWKBZAQPR7NOM3PFIOR3L75WUAJ2WDSDFLTUOHWPEFQIXL3A, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.780054686 (2h30m51.560988072s) [noise=true , address=45.91.116.12:4226 , piece_id=Z7BOS5IU7OH3LJK5FR544QNNKBICQKUM6B7OLNUZKBFNT3VLHNVA, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.781269803 (2h30m51.562203189s) [noise=true , address=45.91.116.12:4227 , piece_id=VBWEJWG2HL5HASAM2O3SJ5BLO56EAVKKCZ7MUZZCP4WTHVKQQ47A, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.781796509 (2h30m51.562729898s) [noise=true , address=45.91.116.13:4155 , piece_id=DXQDYFKUQHC5SYOOT7MANPI5U6YZAF5AASTQWTZTWEWW4UYGCNEQ, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.782987131 (2h30m51.56392052s) [noise=true , address=77.90.31.10:4518 , piece_id=IAWAZEPPEVPCCNNXU5Y5RGWGMZBFBRZNHD22DAV5JUOKUIXW6VLQ, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.784188079 (2h30m51.565121467s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRWSPQB4KHTOPRSJTSIIKABELTEHJUSXKO6OTPNPJJK2DVWKO5A, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.785505146 (2h30m51.566438526s) [noise=true , address=45.91.116.6:4974 , piece_id=Q3MDAYIJEVD4HMBOMJWJPVZRNN62AMPVOS7N3Z4C5CVM74CG6UTQ, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.786530557 (2h30m51.567463943s) [noise=true , address=45.91.116.9:4262 , piece_id=X6LWIEQ5NSGSKGJ3WGHQABIDCNDDLAEMZ7BKBYHHIPMEAKVRPVGA, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.787095609 (2h30m51.568029008s) [noise=true , address=45.91.116.5:4298 , piece_id=GSLSFIKGIYWEIAECQWBL6PSPT7BPKRK2SV4WSFKMHRX7C3Z4VAYA, seg_pos={0 0}, upload=2316]: Uploading piece... Jul 27 00:22:35.868798021 (2h30m51.649731401s) [upload=2316]: Next returned nil segment Jul 27 00:22:35.868817824 (2h30m51.649751208s) [upload=2316]: Waiting for error group managing segments... Jul 27 00:22:35.873835790 (2h30m51.654769171s) [noise=true , address=45.91.116.5:4298 , piece_id=GSLSFIKGIYWEIAECQWBL6PSPT7BPKRK2SV4WSFKMHRX7C3Z4VAYA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.874532930 (2h30m51.655466307s) [noise=true , address=45.91.116.12:4227 , piece_id=VBWEJWG2HL5HASAM2O3SJ5BLO56EAVKKCZ7MUZZCP4WTHVKQQ47A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.875012943 (2h30m51.655946312s) [noise=true , address=45.91.116.11:4119 , piece_id=V4BNG3X6KOITLDNCFWT6XBDZOJCYDXO5H3N6AJ7MKKF4NXMP72AQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.875470060 (2h30m51.65640343s) [noise=true , address=45.91.116.6:4974 , piece_id=Q3MDAYIJEVD4HMBOMJWJPVZRNN62AMPVOS7N3Z4C5CVM74CG6UTQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.875854097 (2h30m51.656787466s) [noise=true , address=45.91.116.12:4226 , piece_id=Z7BOS5IU7OH3LJK5FR544QNNKBICQKUM6B7OLNUZKBFNT3VLHNVA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.876566577 (2h30m51.657499958s) [noise=true , address=45.91.116.13:4155 , piece_id=DXQDYFKUQHC5SYOOT7MANPI5U6YZAF5AASTQWTZTWEWW4UYGCNEQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.877190659 (2h30m51.658124035s) [noise=true , address=45.91.116.10:4082 , piece_id=OA25WWKBZAQPR7NOM3PFIOR3L75WUAJ2WDSDFLTUOHWPEFQIXL3A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.877759205 (2h30m51.658692586s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRWSPQB4KHTOPRSJTSIIKABELTEHJUSXKO6OTPNPJJK2DVWKO5A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.878468274 (2h30m51.659401638s) [noise=true , address=77.90.31.14:4663 , piece_id=S36NSRGS7IGRKBIOKHLZGMZLIKMXZRSCVZS5WJZIIP7FLGONKLHA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:22:35.904448561 (2h30m51.685381938s) [noise=true , address=45.91.116.9:4262 , piece_id=X6LWIEQ5NSGSKGJ3WGHQABIDCNDDLAEMZ7BKBYHHIPMEAKVRPVGA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil> Jul 27 00:23:05.695637857 (2h31m21.476571232s) [noise=true , address=77.90.31.10:4517 , piece_id=4TEO72K4T5IBB3KAYDJ7XA3BMPSOIX4FGLOSM3AYBDVSKRVPGX7A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:05.695721000 (2h31m21.476654377s) [noise=true , address=104.36.82.14:4223 , piece_id=364455LQ6FYYIA6II4JS7PKFZM7422U2B2KD4DD3VGYVJPQ2YZWA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:05.695783323 (2h31m21.476716699s) [noise=true , address=77.90.31.10:4518 , piece_id=IAWAZEPPEVPCCNNXU5Y5RGWGMZBFBRZNHD22DAV5JUOKUIXW6VLQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:05.695822693 (2h31m21.476756068s) [noise=true , address=104.36.82.16:4295 , piece_id=3CNUVGW4XWZB4PMYKNWVKHKNRQH5M4XYWLULZ6HPY4GYBJBUYWIQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:05.695836853 (2h31m21.476770228s) [noise=true , address=104.36.82.2:4731 , piece_id=TAHBPPGVULUOM3KNMPUWBGLHUJMETCHHBPWZSP5VRFG3GSTEZSGA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:05.695874418 (2h31m21.476807783s) [noise=true , address=77.90.31.21:4878 , piece_id=27RVSTANPTACCVI7UO2LIM2K46SIGQX4A47X7AHKZP2KQPK5XHUA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:42704->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:05.695985434 (2h31m21.476918815s) [seg_pos={0 0}, upload=2316]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:05.719386464 (2h31m21.500319826s) [upload=2316]: Done uploading segments... Jul 27 00:23:05.719418120 (2h31m21.500351499s) [upload=2316]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.02618536s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........X.X........X...XX...............................X] duration: 1m0.002104184s FAIL (2025-07-27 00:22:56 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002152392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXX...............X..X......X.XX....X...X.................X] duration: 30.000767384s FAIL (2025-07-27 00:22:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0008631s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.X] duration: 30.027159506s FAIL (2025-07-27 00:22:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.027191673s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032764142s FAIL (2025-07-27 00:22:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== BatchRequestItem_SegmentBegin Jul 27 00:21:25.845611332 (2h29m41.626544712s) [seg_pos={0 0}, upload=2299]: Begin upload segment... Jul 27 00:21:25.845681099 (2h29m41.626614467s) [seg_pos={0 0}, upload=2299]: Done begin upload segment. Jul 27 00:21:25.845710878 (2h29m41.626644259s) [noise=true , address=79.127.205.234:4607 , piece_id=B54J4QSM5AE5GDEKOKBO34G5ZDXKN2SKQU6NG5VYOV62TGMMSZIA, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.845740303 (2h29m41.626673688s) [noise=true , address=77.90.31.16:4759 , piece_id=5NR4O3SPBDGEPESQ7CMTO5EMB3FINRPHSHBPKG2HVLWDJOUEDPMQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.846239235 (2h29m41.627172615s) [noise=true , address=79.127.201.215:4346 , piece_id=JWCVVLU6WF6DIFSO7QZ3MJVV3E55PXSW7KLWYA5Y7R5WU55VC4XQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.846317932 (2h29m41.627251318s) [noise=true , address=79.127.201.214:4289 , piece_id=BE3AO3IK2A4EITAPW2AQ3ZI57OA3PXMGAKTQ2FIFAK6VKLOHXP2A, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.846656216 (2h29m41.627589623s) [noise=true , address=104.36.82.55:4201 , piece_id=AFLD4XESUM7DQARC7URD62VSUK5XBNWOQ2VBAAVHQ72GSSD3SNVA, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.846842935 (2h29m41.627776317s) [noise=true , address=104.36.82.56:4260 , piece_id=2BV2FFWULCV6UVYYI5G3RNWTGLWCKNAFLHWW7VPRQVBRNXKMDSFQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.847082090 (2h29m41.628015504s) [noise=true , address=79.127.205.233:4655 , piece_id=B6R6YUL37ETSDZRW5WWVIP2NMAFDRHFZ6MENEUEQ3JOIV26LTEIQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.847389221 (2h29m41.628322601s) [noise=true , address=77.90.31.10:4531 , piece_id=XUC6OOB3CNEG7JRLVHLJ7EK64NDVDCL723XIAI7SHX76P6JCBUFQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.847529182 (2h29m41.628462569s) [noise=true , address=156.146.43.225:4448 , piece_id=Z6ZQXQR3GLMXQKYS3RHNTCCGJXFLUW2GB5BZM6VHVHKJFD46MQMA, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.847864818 (2h29m41.628798207s) [noise=true , address=104.36.82.4:4832 , piece_id=LQXHJXDCFOH5VOJUDU5HZBHJLXW467LCBR55C4ZDARJIOR672ZQQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.848089684 (2h29m41.629023066s) [noise=true , address=64.49.13.140:4508 , piece_id=MB3PMTNGBQJLEORLRX5BE4DJBEUXSUGU5D4H63JGFJGXBSZYUOOA, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.848411645 (2h29m41.629345029s) [noise=true , address=77.90.31.13:4656 , piece_id=R4OVOCMZ6FPZRMSVTPLHXKPXY5DSSKFPTBJFQL4PRXG37AIPG62A, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.848460712 (2h29m41.6293941s) [noise=true , address=64.49.13.138:4483 , piece_id=YQTUONU3E3SF5ZTFD7RKZEGWOPVIWTKYK3UD3O3ONNFKETA6L2DA, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.848886460 (2h29m41.629819842s) [noise=true , address=64.49.13.26:4953 , piece_id=5HFKVX5V7KHR7Z7MMIHQ2DJHBDOHU6ONEZHKZMZKKQ7XLCRNES5A, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.849374433 (2h29m41.630307817s) [noise=true , address=104.36.82.47:4543 , piece_id=SI4VNAMTDUZ4WM33M5HAYYZ2QJATNWOBEBEI3WVOLMRH42E4BUOQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.849853476 (2h29m41.630786858s) [noise=true , address=79.127.205.246:4276 , piece_id=3L5UP6CPVF23R37T6XIXVY4SLZRUSCMBGVCIOTQRR7WM6NN44R6Q, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.850498602 (2h29m41.631431984s) [noise=true , address=104.36.82.52:4110 , piece_id=OCYBKVDGL25E6UPEIEKP5YNC525MRFHWUOPZZUVFBJHY37663ZUQ, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.850688866 (2h29m41.631622302s) [noise=true , address=77.90.31.11:4574 , piece_id=TJ7NTCZF4PUH5CUOTVK56DCSTYGFMAKNYZ5D7MHEFTEJLSWM5MBA, seg_pos={0 0}, upload=2299]: Uploading piece... Jul 27 00:21:25.950770091 (2h29m41.731703477s) [upload=2299]: Next returned nil segment Jul 27 00:21:25.950794342 (2h29m41.731727725s) [upload=2299]: Waiting for error group managing segments... Jul 27 00:21:26.560293150 (2h29m42.341226547s) [noise=true , address=79.127.205.246:4276 , piece_id=3L5UP6CPVF23R37T6XIXVY4SLZRUSCMBGVCIOTQRR7WM6NN44R6Q, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:26.692037765 (2h29m42.472971148s) [noise=true , address=79.127.205.233:4655 , piece_id=B6R6YUL37ETSDZRW5WWVIP2NMAFDRHFZ6MENEUEQ3JOIV26LTEIQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:26.746683600 (2h29m42.527616979s) [noise=true , address=77.90.31.16:4759 , piece_id=5NR4O3SPBDGEPESQ7CMTO5EMB3FINRPHSHBPKG2HVLWDJOUEDPMQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:26.856216536 (2h29m42.637149905s) [noise=true , address=79.127.205.234:4607 , piece_id=B54J4QSM5AE5GDEKOKBO34G5ZDXKN2SKQU6NG5VYOV62TGMMSZIA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:28.719383028 (2h29m44.500316408s) [noise=true , address=156.146.43.225:4448 , piece_id=Z6ZQXQR3GLMXQKYS3RHNTCCGJXFLUW2GB5BZM6VHVHKJFD46MQMA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:28.771688449 (2h29m44.552621814s) [noise=true , address=64.49.13.26:4953 , piece_id=5HFKVX5V7KHR7Z7MMIHQ2DJHBDOHU6ONEZHKZMZKKQ7XLCRNES5A, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:30.552322605 (2h29m46.333255993s) [noise=true , address=79.127.201.214:4289 , piece_id=BE3AO3IK2A4EITAPW2AQ3ZI57OA3PXMGAKTQ2FIFAK6VKLOHXP2A, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:32.747795444 (2h29m48.528728808s) [noise=true , address=64.49.13.140:4508 , piece_id=MB3PMTNGBQJLEORLRX5BE4DJBEUXSUGU5D4H63JGFJGXBSZYUOOA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:38.044431639 (2h29m53.825365025s) [noise=true , address=64.49.13.138:4483 , piece_id=YQTUONU3E3SF5ZTFD7RKZEGWOPVIWTKYK3UD3O3ONNFKETA6L2DA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:21:48.213401071 (2h30m3.994334455s) [noise=true , address=79.127.201.215:4346 , piece_id=JWCVVLU6WF6DIFSO7QZ3MJVV3E55PXSW7KLWYA5Y7R5WU55VC4XQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:22:11.958712268 (2h30m27.739645638s) [noise=true , address=77.90.31.13:4656 , piece_id=R4OVOCMZ6FPZRMSVTPLHXKPXY5DSSKFPTBJFQL4PRXG37AIPG62A, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil> Jul 27 00:22:25.772740938 (2h30m41.553674306s) [noise=true , address=104.36.82.52:4110 , piece_id=OCYBKVDGL25E6UPEIEKP5YNC525MRFHWUOPZZUVFBJHY37663ZUQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:25.772816418 (2h30m41.553749784s) [noise=true , address=104.36.82.55:4201 , piece_id=AFLD4XESUM7DQARC7URD62VSUK5XBNWOQ2VBAAVHQ72GSSD3SNVA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:25.772885980 (2h30m41.553819347s) [noise=true , address=104.36.82.47:4543 , piece_id=SI4VNAMTDUZ4WM33M5HAYYZ2QJATNWOBEBEI3WVOLMRH42E4BUOQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: write tcp 159.223.143.39:50240->104.36.82.47:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:25.773084199 (2h30m41.554017561s) [noise=true , address=104.36.82.56:4260 , piece_id=2BV2FFWULCV6UVYYI5G3RNWTGLWCKNAFLHWW7VPRQVBRNXKMDSFQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:25.773110483 (2h30m41.554043868s) [noise=true , address=77.90.31.10:4531 , piece_id=XUC6OOB3CNEG7JRLVHLJ7EK64NDVDCL723XIAI7SHX76P6JCBUFQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: rpc: dial tcp 77.90.31.10:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:25.773216295 (2h30m41.554149663s) [noise=true , address=104.36.82.4:4832 , piece_id=LQXHJXDCFOH5VOJUDU5HZBHJLXW467LCBR55C4ZDARJIOR672ZQQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 159.223.143.39:52510->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:25.773249204 (2h30m41.554182587s) [noise=true , address=77.90.31.11:4574 , piece_id=TJ7NTCZF4PUH5CUOTVK56DCSTYGFMAKNYZ5D7MHEFTEJLSWM5MBA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:25.773313587 (2h30m41.554246972s) [seg_pos={0 0}, upload=2299]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:22:25.802897449 (2h30m41.583830828s) [upload=2299]: Done uploading segments... Jul 27 00:22:25.802931970 (2h30m41.583865347s) [upload=2299]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.032787184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XX] duration: 30.027886622s FAIL (2025-07-27 00:22:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:57.006580930 (2h29m12.787514304s) [noise=true , address=77.90.31.21:4877 , piece_id=67JFI32F7BX2ZNADJPNMT4XWE3P6IRNYNWEKZV3VUF5KQRAPP2MQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:57.006650182 (2h29m12.787583572s) [seg_pos={0 0}, upload=2287]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:20:57.034251290 (2h29m12.815184674s) [upload=2287]: Done uploading segments... Jul 27 00:20:57.034293667 (2h29m12.815227031s) [upload=2287]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:21:37.028547529 (2h29m52.809480919s) [upload=2302]: Starting upload Jul 27 00:21:37.028652520 (2h29m52.809585908s) [upload=2302]: Done starting upload Jul 27 00:21:37.028754843 (2h29m52.809688226s) [upload=2302]: Uploading segments... Jul 27 00:21:37.029190281 (2h29m52.81012367s) [upload=2302]: Got next segment. Inline: false Jul 27 00:21:37.029248393 (2h29m52.810181785s) [upload=2302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:21:37.029256592 (2h29m52.810189978s) [upload=2302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:21:37.060493438 (2h29m52.841426836s) [seg_pos={0 0}, upload=2302]: Begin upload segment... Jul 27 00:21:37.060553105 (2h29m52.841486487s) [seg_pos={0 0}, upload=2302]: Done begin upload segment. Jul 27 00:21:37.060585770 (2h29m52.841519158s) [noise=true , address=104.36.82.14:4223 , piece_id=NYJ2SS7HIAJCBUI5D4LWUA5QSTDBO6QDNEPYAUEUCHGCJNJRJLFQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.061114396 (2h29m52.842047781s) [noise=true , address=45.91.116.11:4117 , piece_id=AG5TRPYVVSW4OIOKNBGWJ4YGA5KTHZV4PZBOCG4MESFULBX4VPZQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.061522230 (2h29m52.84245562s) [noise=true , address=104.36.82.16:4295 , piece_id=JMN6GMFWIATKVDZYJOFEDHQS4JUPW7ZIW7C4N6RKU7UUXD7CU3IQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.061911594 (2h29m52.842844983s) [noise=true , address=45.91.116.10:4083 , piece_id=FMBUNIBVIYKSKXHX5R2QGRN224ISWS4KXGN7EKEZ2PUIZISWFEKA, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.062297035 (2h29m52.843230418s) [noise=true , address=45.91.116.8:4045 , piece_id=VEKBCCJBSXNK5ZRV4UBJ2Z2X6X6JP5D7B7TB7PCG477OMTC7ZNEQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.062687955 (2h29m52.843621339s) [noise=true , address=45.91.116.13:4154 , piece_id=DVEQGDU36AC3TL4WX77R7BNPJ52QWH5PHK6YFGTWCOAPWOMLU3KA, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.063098303 (2h29m52.844031691s) [noise=true , address=45.91.116.12:4225 , piece_id=SPUQTI5EMC7DM3DEUTTC7X4ZOORKHSMKYMD67P4SORRJ5TWO3TDQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.063483303 (2h29m52.844416687s) [noise=true , address=45.91.116.4:4937 , piece_id=6MM2OUY57MADCYUBSODXLBZ6Q62NIPR2GY2XYQIBYJVHPLKNEBMA, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.063952392 (2h29m52.844885788s) [noise=true , address=104.36.82.8:4971 , piece_id=KSKXDPTXCJMZKL3JBL5QRNQIJ2X7ZLULFU62CQPP6LNRT5KBUJSA, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.064322978 (2h29m52.845256363s) [noise=true , address=104.36.82.6:4873 , piece_id=EURLWODJCUALWBRTOLZLY646L7W2TDEOB45YVJJY66RHP5OH4MRA, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.064665253 (2h29m52.845598618s) [noise=true , address=45.91.116.14:4191 , piece_id=OBZWTD6J4ZWY252WY6LHK3RBEO3WKPQGHDN2MWDDIRVLWA5RHKFQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.065010995 (2h29m52.845944379s) [noise=true , address=77.90.31.14:4661 , piece_id=KCPPL3E27PGFJRZLDQ5V3BWPYTRAERHQLWIDX6KOABOS4NBR2JWQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.065416498 (2h29m52.846349899s) [noise=true , address=45.91.116.9:4261 , piece_id=BO7BHA3ZWYI7UNFMJG6R3OCEZFKKRDZP3FVCXR3FAIDWF34DZRMA, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.065784904 (2h29m52.84671829s) [noise=true , address=45.91.116.7:4009 , piece_id=JQHUZYFLLL2SHUNVBC2MLYBUK6OVJYA3R73OC5HINV4C7TDVKJCA, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.066256381 (2h29m52.847189881s) [noise=true , address=45.91.116.6:4974 , piece_id=ZZOAL4EUDFJ7WOAPAIDGP24FFNB25VG7K3W7365LYROVG3MGHXCQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.066660270 (2h29m52.847593656s) [noise=true , address=77.90.31.16:4735 , piece_id=RZJRH6QHIEE6CMIX32FUIOPYLKANZH54JJLOXSPUEH7Z4LJXY4NQ, seg_pos={0 0}, upload=2302]: Uploading piece... Jul 27 00:21:37.167427333 (2h29m52.948360715s) [upload=2302]: Next returned nil segment Jul 27 00:21:37.167453250 (2h29m52.948386644s) [upload=2302]: Waiting for error group managing segments... Jul 27 00:21:37.171892029 (2h29m52.952825407s) [noise=true , address=45.91.116.13:4154 , piece_id=DVEQGDU36AC3TL4WX77R7BNPJ52QWH5PHK6YFGTWCOAPWOMLU3KA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.172684508 (2h29m52.953617888s) [noise=true , address=45.91.116.12:4225 , piece_id=SPUQTI5EMC7DM3DEUTTC7X4ZOORKHSMKYMD67P4SORRJ5TWO3TDQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.173263631 (2h29m52.954196996s) [noise=true , address=45.91.116.7:4009 , piece_id=JQHUZYFLLL2SHUNVBC2MLYBUK6OVJYA3R73OC5HINV4C7TDVKJCA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.173867164 (2h29m52.954800536s) [noise=true , address=77.90.31.14:4661 , piece_id=KCPPL3E27PGFJRZLDQ5V3BWPYTRAERHQLWIDX6KOABOS4NBR2JWQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.174384305 (2h29m52.955317673s) [noise=true , address=45.91.116.9:4261 , piece_id=BO7BHA3ZWYI7UNFMJG6R3OCEZFKKRDZP3FVCXR3FAIDWF34DZRMA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.174885307 (2h29m52.955818684s) [noise=true , address=45.91.116.4:4937 , piece_id=6MM2OUY57MADCYUBSODXLBZ6Q62NIPR2GY2XYQIBYJVHPLKNEBMA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.175399043 (2h29m52.956332424s) [noise=true , address=45.91.116.8:4045 , piece_id=VEKBCCJBSXNK5ZRV4UBJ2Z2X6X6JP5D7B7TB7PCG477OMTC7ZNEQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.175908678 (2h29m52.956842045s) [noise=true , address=45.91.116.11:4117 , piece_id=AG5TRPYVVSW4OIOKNBGWJ4YGA5KTHZV4PZBOCG4MESFULBX4VPZQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.176528987 (2h29m52.957462351s) [noise=true , address=45.91.116.10:4083 , piece_id=FMBUNIBVIYKSKXHX5R2QGRN224ISWS4KXGN7EKEZ2PUIZISWFEKA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.176825269 (2h29m52.957758638s) [noise=true , address=45.91.116.6:4974 , piece_id=ZZOAL4EUDFJ7WOAPAIDGP24FFNB25VG7K3W7365LYROVG3MGHXCQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:37.212768672 (2h29m52.993702053s) [noise=true , address=45.91.116.14:4191 , piece_id=OBZWTD6J4ZWY252WY6LHK3RBEO3WKPQGHDN2MWDDIRVLWA5RHKFQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:21:38.615196614 (2h29m54.396129978s) [noise=true , address=77.90.31.16:4735 , piece_id=RZJRH6QHIEE6CMIX32FUIOPYLKANZH54JJLOXSPUEH7Z4LJXY4NQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil> Jul 27 00:22:06.992686366 (2h30m22.773619762s) [noise=true , address=104.36.82.8:4971 , piece_id=KSKXDPTXCJMZKL3JBL5QRNQIJ2X7ZLULFU62CQPP6LNRT5KBUJSA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:44652->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:06.992787636 (2h30m22.773721013s) [noise=true , address=104.36.82.16:4295 , piece_id=JMN6GMFWIATKVDZYJOFEDHQS4JUPW7ZIW7C4N6RKU7UUXD7CU3IQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:06.992854441 (2h30m22.773787819s) [noise=true , address=104.36.82.6:4873 , piece_id=EURLWODJCUALWBRTOLZLY646L7W2TDEOB45YVJJY66RHP5OH4MRA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:48040->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:06.992919905 (2h30m22.773853268s) [noise=true , address=104.36.82.14:4223 , piece_id=NYJ2SS7HIAJCBUI5D4LWUA5QSTDBO6QDNEPYAUEUCHGCJNJRJLFQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:58496->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:06.992951239 (2h30m22.773884626s) [seg_pos={0 0}, upload=2302]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:22:07.017432150 (2h30m22.798365536s) [upload=2302]: Done uploading segments... Jul 27 00:22:07.017466365 (2h30m22.79839975s) [upload=2302]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027913004s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X.X........X...XX...............................] duration: 41.358878973s FAIL (2025-07-27 00:21:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.358896966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.02982462s FAIL (2025-07-27 00:21:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== oading piece... Jul 27 00:20:25.750008228 (2h28m41.530941609s) [noise=true , address=104.36.82.52:4103 , piece_id=7XF3HHNNUEUM2XT7U72JN4ELTUSM4IJ7AQTWNKVOPBDAO7NG3VZA, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.750398065 (2h28m41.531331448s) [noise=true , address=79.127.205.245:4244 , piece_id=NZPKZPY6DRVJCLQEHMSLQGCGVJMSH4X75KQPQFU6522N3DX4KJTQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.750413229 (2h28m41.531346617s) [noise=true , address=104.36.82.15:4270 , piece_id=UXNWIMSPDEYXQCC6GVZRBNHFHLEYNIBCEFMUQFZ3IEEZIDFPPOFA, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.750942148 (2h28m41.531875532s) [noise=true , address=104.36.82.44:4874 , piece_id=YA7TS3PFJUES3YOS3S4PJTV6IO6TRTSNRUXNQ3JNA6Y22LDASQRA, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.751838164 (2h28m41.532771546s) [noise=true , address=79.127.205.234:4613 , piece_id=AUPGDA24IXGQCY7BJPBHGT3IKHUGN5QKQRTL2DUIBK2IGDUBIJPA, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.752463952 (2h28m41.53339733s) [noise=true , address=79.127.205.235:4685 , piece_id=XTQG6BM3UDR5GPOFDBSJ754ZFV6QUMZFGIJWN6UJYCOJSGPVGRSQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.752947463 (2h28m41.533880849s) [noise=true , address=79.127.201.215:4346 , piece_id=V4LS32XF67KJETKZDNWEZQTBHGFGJUAN3CRCCJLMW7XMS2ALOVZQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.753452630 (2h28m41.534386019s) [noise=true , address=104.36.82.54:4183 , piece_id=J6HARDQZWX77R3HN3W435AFD3HLSWQO4IADDFXPYL5V6GAOVYJTQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.753885872 (2h28m41.534819258s) [noise=true , address=104.36.82.47:4548 , piece_id=4XZC5P4LMAMYUGCXDV53VPPSJE2LHWHDPGVLB2YLPDWD2UR6JWVA, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.754327743 (2h28m41.53526113s) [noise=true , address=77.90.31.16:4751 , piece_id=KQVRGH5PABRYJEM2ZCINO6WDOTWYDKB3AH3VB765MAWN7XVBIBWA, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.754848598 (2h28m41.535781983s) [noise=true , address=104.36.82.56:4260 , piece_id=AASN7ZRQJBVEFBL5OQSVGRXXNN52IJZZ4WMJMFFX6QKM6G3R7MBQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.755341568 (2h28m41.53627495s) [noise=true , address=77.90.31.19:4847 , piece_id=WZ6J5BIHGKJTVBKFUHNS7Q6NGIMJMA6WKPVQUKCGB36MUCRIK4TQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.755893713 (2h28m41.536827096s) [noise=true , address=104.36.82.46:4509 , piece_id=5L6LGGFXDTYNYIPB7ZXZAWXZHC35WEVBOOELVOGD4YFBQS2KMQOQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.756309800 (2h28m41.537243184s) [noise=true , address=77.90.31.11:4574 , piece_id=WQUYLBLMJSEJBUN6OYGXH2WYJ3T2ANBTZEHNWMPHH2ZXCCG4SKTQ, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.756677341 (2h28m41.537610726s) [noise=true , address=156.146.43.225:4448 , piece_id=2HGVZSO4GOKDJOVHXE2MN4MAMW26WLA6ZM6LWN4EBGC6IHI2UEXA, seg_pos={0 0}, upload=2286]: Uploading piece... Jul 27 00:20:25.833891044 (2h28m41.614824416s) [upload=2286]: Next returned nil segment Jul 27 00:20:25.833914457 (2h28m41.614847853s) [upload=2286]: Waiting for error group managing segments... Jul 27 00:20:26.124753455 (2h28m41.905686842s) [noise=true , address=64.49.13.140:4514 , piece_id=QUAVPSXBHZXPI5WMUXPIVIHVVN7J24SA4TSPADVV4YONM7KKFTUQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 27 00:20:26.361148451 (2h28m42.142081829s) [noise=true , address=79.127.205.234:4613 , piece_id=AUPGDA24IXGQCY7BJPBHGT3IKHUGN5QKQRTL2DUIBK2IGDUBIJPA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil> Jul 27 00:20:26.406600519 (2h28m42.187533909s) [noise=true , address=79.127.205.235:4685 , piece_id=XTQG6BM3UDR5GPOFDBSJ754ZFV6QUMZFGIJWN6UJYCOJSGPVGRSQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil> Jul 27 00:20:26.609224146 (2h28m42.390157526s) [noise=true , address=79.127.205.245:4244 , piece_id=NZPKZPY6DRVJCLQEHMSLQGCGVJMSH4X75KQPQFU6522N3DX4KJTQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil> Jul 27 00:20:26.678160553 (2h28m42.45909392s) [noise=true , address=79.127.201.215:4346 , piece_id=V4LS32XF67KJETKZDNWEZQTBHGFGJUAN3CRCCJLMW7XMS2ALOVZQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil> Jul 27 00:20:27.166649857 (2h28m42.947583239s) [noise=true , address=156.146.43.225:4448 , piece_id=2HGVZSO4GOKDJOVHXE2MN4MAMW26WLA6ZM6LWN4EBGC6IHI2UEXA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil> Jul 27 00:20:29.922900751 (2h28m45.70383412s) [noise=true , address=77.90.31.16:4751 , piece_id=KQVRGH5PABRYJEM2ZCINO6WDOTWYDKB3AH3VB765MAWN7XVBIBWA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil> Jul 27 00:20:38.820722829 (2h28m54.601656199s) [noise=true , address=77.90.31.19:4847 , piece_id=WZ6J5BIHGKJTVBKFUHNS7Q6NGIMJMA6WKPVQUKCGB36MUCRIK4TQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil> Jul 27 00:21:25.681480878 (2h29m41.462414257s) [noise=true , address=104.36.82.56:4260 , piece_id=AASN7ZRQJBVEFBL5OQSVGRXXNN52IJZZ4WMJMFFX6QKM6G3R7MBQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681572325 (2h29m41.462505703s) [noise=true , address=104.36.82.46:4509 , piece_id=5L6LGGFXDTYNYIPB7ZXZAWXZHC35WEVBOOELVOGD4YFBQS2KMQOQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: rpc: dial tcp 104.36.82.46:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681646155 (2h29m41.462579533s) [noise=true , address=104.36.82.55:4201 , piece_id=H43645BI3ZAW6HOOGO4YINQMNW2KXW7RANSQVV2WGW7765IBSQUA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681703115 (2h29m41.462636493s) [noise=true , address=104.36.82.54:4183 , piece_id=J6HARDQZWX77R3HN3W435AFD3HLSWQO4IADDFXPYL5V6GAOVYJTQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681761630 (2h29m41.462695007s) [noise=true , address=77.90.31.11:4574 , piece_id=WQUYLBLMJSEJBUN6OYGXH2WYJ3T2ANBTZEHNWMPHH2ZXCCG4SKTQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:36176->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681806821 (2h29m41.462740201s) [noise=true , address=104.36.82.44:4874 , piece_id=YA7TS3PFJUES3YOS3S4PJTV6IO6TRTSNRUXNQ3JNA6Y22LDASQRA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq, address:104.36.82.44:4874): protocol: rpc: dial tcp 104.36.82.44:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681843169 (2h29m41.462776547s) [noise=true , address=104.36.82.47:4548 , piece_id=4XZC5P4LMAMYUGCXDV53VPPSJE2LHWHDPGVLB2YLPDWD2UR6JWVA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681915516 (2h29m41.462848894s) [noise=true , address=104.36.82.42:4816 , piece_id=ZRTNJ5HFG2O2TGDNJMISOCQBTZF6AWZ3JPTSDYQA5JCFVL7BKD7Q, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1nq6pTeLCMEmtagczJkTMAdpADG5JrbSKhUjXAHy9oeS1pTpSx, address:104.36.82.42:4816): protocol: rpc: dial tcp 104.36.82.42:4816: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.681938875 (2h29m41.462872246s) [noise=true , address=104.36.82.52:4103 , piece_id=7XF3HHNNUEUM2XT7U72JN4ELTUSM4IJ7AQTWNKVOPBDAO7NG3VZA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.682014743 (2h29m41.462948114s) [noise=true , address=104.36.82.15:4270 , piece_id=UXNWIMSPDEYXQCC6GVZRBNHFHLEYNIBCEFMUQFZ3IEEZIDFPPOFA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: write tcp 159.223.143.39:40398->104.36.82.15:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:21:25.682067892 (2h29m41.463001279s) [seg_pos={0 0}, upload=2286]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:21:25.709020105 (2h29m41.489953476s) [upload=2286]: Done uploading segments... Jul 27 00:21:25.709070623 (2h29m41.490004005s) [upload=2286]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.02984712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.XXX...............X..X......X.XX....X...X.................] duration: 2.589557223s FAIL (2025-07-27 00:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.589589532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.] duration: 2.443810874s FAIL (2025-07-27 00:20:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 2.443859293s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....X] duration: 30.03066556s FAIL (2025-07-27 00:20:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== k connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:41.004485343 (2h27m56.785418722s) [seg_pos={0 0}, upload=2273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:19:41.004577727 (2h27m56.785511106s) [upload=2273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:19:41.004584582 (2h27m56.785517962s) [upload=2273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:19:41.056659105 (2h27m56.83759247s) [upload=2273]: Done uploading segments... Jul 27 00:19:41.056683745 (2h27m56.837617107s) [upload=2273]: Upload finished. err: <nil> Jul 27 00:20:27.035648791 (2h28m42.816582178s) [upload=2287]: Starting upload Jul 27 00:20:27.035802988 (2h28m42.816736349s) [upload=2287]: Done starting upload Jul 27 00:20:27.036057893 (2h28m42.81699126s) [upload=2287]: Uploading segments... Jul 27 00:20:27.036395184 (2h28m42.817328559s) [upload=2287]: Got next segment. Inline: false Jul 27 00:20:27.036466520 (2h28m42.817399896s) [upload=2287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:20:27.036475002 (2h28m42.817408382s) [upload=2287]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:20:27.066900323 (2h28m42.847833705s) [seg_pos={0 0}, upload=2287]: Begin upload segment... Jul 27 00:20:27.066956201 (2h28m42.847889583s) [seg_pos={0 0}, upload=2287]: Done begin upload segment. Jul 27 00:20:27.066990267 (2h28m42.84792366s) [noise=true , address=45.91.116.7:4009 , piece_id=C6UBSKL6EUJKUHGBV3B3CV5KOVARUAOOVZWXK3HUQDPDEW3UF6PQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.067074462 (2h28m42.848007846s) [noise=true , address=77.90.31.18:4807 , piece_id=PPTKCS3RZFAF5O7LB66WYY7KGREPDZ5KU55QZHUF4M72DYCDNTQQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.067470705 (2h28m42.848404098s) [noise=true , address=45.91.116.6:4975 , piece_id=BEDHQWJQ5HVFDVI3GYQL2T33YGNPFCLRZYKXZHXRY56O4CIN277Q, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.067643399 (2h28m42.848576785s) [noise=true , address=77.90.31.11:4553 , piece_id=4NQHV2AWCLHCH2H7OZ4OWTGIHZFGTGLEUAKULOE3VB7TJFJ3DXHQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.067915059 (2h28m42.848848443s) [noise=true , address=45.91.116.11:4119 , piece_id=GFXD2IHI6EIM4ZFNEWTEQKRXCQJT5UDPOJ4DNZT7P6JYT54354EA, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.068285088 (2h28m42.849218476s) [noise=true , address=45.91.116.8:4045 , piece_id=K4KJEGN32V674LIOFRBDOVHC2MBMDE7MG6EXDULE2SKNHGCSPTCQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.068446646 (2h28m42.849380041s) [noise=true , address=45.91.116.4:4939 , piece_id=IDRX5BRF4KGKWHZLESPCCTNV6VNPFMN7AFOKLCN7HSNZGISJFONA, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.068773204 (2h28m42.849706584s) [noise=true , address=77.90.31.14:4663 , piece_id=KUVPM5352PFA6CNYHZQWPKRPBMSG5Q5JHAZUKVR7TE3JIOAOL4OA, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.068834037 (2h28m42.849767424s) [noise=true , address=45.91.116.12:4225 , piece_id=FZKCX7WR2Y2H2V2AFGHV2JMUPAI2PN6HV2PADQKOYQR7KRHAIE5Q, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.069101011 (2h28m42.850034399s) [noise=true , address=77.90.31.21:4877 , piece_id=67JFI32F7BX2ZNADJPNMT4XWE3P6IRNYNWEKZV3VUF5KQRAPP2MQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.069390923 (2h28m42.850324317s) [noise=true , address=104.36.82.4:4802 , piece_id=7IJO6A2ZCUFYMMTTJJXLZDXKXMSTKDDQA3M6G6OHTF7BMN65PUTQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.069417578 (2h28m42.850350958s) [noise=true , address=104.36.82.9:4006 , piece_id=YTL6SHGOFZ4QGPVKLAIZQOD26S23LK3DQTPFBYP54WAO7QSMCAAQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.069706097 (2h28m42.850639476s) [noise=true , address=104.36.82.16:4294 , piece_id=CJOUPJLI7ICUZPCOPTPCYTOZVBAIHEZOYH3O33PCWTGS2UIRLZ7A, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.069955506 (2h28m42.850888893s) [noise=true , address=104.36.82.16:4293 , piece_id=APXIGGCECRH2KEM5B7OI7KDWNP43K5VGGLQEPEUTZQYCYZSTCHSA, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.070498452 (2h28m42.851431831s) [noise=true , address=45.91.116.9:4263 , piece_id=56D2LMCWEMUQB7BW5AV75ZB4EAIWFWUXW2ASMPA6V64UEIVI6OVQ, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.071135645 (2h28m42.85206909s) [noise=true , address=45.91.116.13:4155 , piece_id=QQALYGTDCSJPM7ZPS3BKWIQHBQPEI5XGJOV4ULFDMWGJ7QSF4T6Q, seg_pos={0 0}, upload=2287]: Uploading piece... Jul 27 00:20:27.170875724 (2h28m42.951809111s) [upload=2287]: Next returned nil segment Jul 27 00:20:27.170898320 (2h28m42.95183171s) [upload=2287]: Waiting for error group managing segments... Jul 27 00:20:27.174860754 (2h28m42.955794136s) [noise=true , address=45.91.116.11:4119 , piece_id=GFXD2IHI6EIM4ZFNEWTEQKRXCQJT5UDPOJ4DNZT7P6JYT54354EA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.175748787 (2h28m42.956682163s) [noise=true , address=45.91.116.12:4225 , piece_id=FZKCX7WR2Y2H2V2AFGHV2JMUPAI2PN6HV2PADQKOYQR7KRHAIE5Q, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.176640849 (2h28m42.957574228s) [noise=true , address=45.91.116.8:4045 , piece_id=K4KJEGN32V674LIOFRBDOVHC2MBMDE7MG6EXDULE2SKNHGCSPTCQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.176806409 (2h28m42.957739778s) [noise=true , address=77.90.31.18:4807 , piece_id=PPTKCS3RZFAF5O7LB66WYY7KGREPDZ5KU55QZHUF4M72DYCDNTQQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.177168999 (2h28m42.95810238s) [noise=true , address=77.90.31.14:4663 , piece_id=KUVPM5352PFA6CNYHZQWPKRPBMSG5Q5JHAZUKVR7TE3JIOAOL4OA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.177386831 (2h28m42.958320201s) [noise=true , address=45.91.116.9:4263 , piece_id=56D2LMCWEMUQB7BW5AV75ZB4EAIWFWUXW2ASMPA6V64UEIVI6OVQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.177693567 (2h28m42.958626953s) [noise=true , address=45.91.116.4:4939 , piece_id=IDRX5BRF4KGKWHZLESPCCTNV6VNPFMN7AFOKLCN7HSNZGISJFONA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.177951420 (2h28m42.958884796s) [noise=true , address=45.91.116.6:4975 , piece_id=BEDHQWJQ5HVFDVI3GYQL2T33YGNPFCLRZYKXZHXRY56O4CIN277Q, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.178240212 (2h28m42.959173595s) [noise=true , address=45.91.116.7:4009 , piece_id=C6UBSKL6EUJKUHGBV3B3CV5KOVARUAOOVZWXK3HUQDPDEW3UF6PQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:27.178506692 (2h28m42.959440062s) [noise=true , address=45.91.116.13:4155 , piece_id=QQALYGTDCSJPM7ZPS3BKWIQHBQPEI5XGJOV4ULFDMWGJ7QSF4T6Q, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil> Jul 27 00:20:57.006233759 (2h29m12.78716714s) [noise=true , address=104.36.82.9:4006 , piece_id=YTL6SHGOFZ4QGPVKLAIZQOD26S23LK3DQTPFBYP54WAO7QSMCAAQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:57.006303901 (2h29m12.787237291s) [noise=true , address=104.36.82.16:4293 , piece_id=APXIGGCECRH2KEM5B7OI7KDWNP43K5VGGLQEPEUTZQYCYZSTCHSA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:57.006352195 (2h29m12.78728557s) [noise=true , address=104.36.82.4:4802 , piece_id=7IJO6A2ZCUFYMMTTJJXLZDXKXMSTKDDQA3M6G6OHTF7BMN65PUTQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:39186->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:57.006369374 (2h29m12.787302744s) [noise=true , address=104.36.82.16:4294 , piece_id=CJOUPJLI7ICUZPCOPTPCYTOZVBAIHEZOYH3O33PCWTGS2UIRLZ7A, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:57.006407241 (2h29m12.787340611s) [noise=true , address=77.90.31.11:4553 , piece_id=4NQHV2AWCLHCH2H7OZ4OWTGIHZFGTGLEUAKULOE3VB7TJFJ3DXHQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:57.006580930 (2h29m12.787514304s) [noise=true , address=77.90.31.21:4877 , piece_id=67JFI32F7BX2ZNADJPNMT4XWE3P6IRNYNWEKZV3VUF5KQRAPP2MQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:57.006650182 (2h29m12.787583572s) [seg_pos={0 0}, upload=2287]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:20:57.034251290 (2h29m12.815184674s) [upload=2287]: Done uploading segments... Jul 27 00:20:57.034293667 (2h29m12.815227031s) [upload=2287]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.030800625s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.XXX...............X..X......X.XX....X...X................] duration: 21.682671438s FAIL (2025-07-27 00:20:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.682706562s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X.X........X...XX..............................] duration: 17.728662562s FAIL (2025-07-27 00:20:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.72868333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X] duration: 30.038491985s FAIL (2025-07-27 00:20:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.038545626s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X.X........X...XX.............................] duration: 38.053913073s FAIL (2025-07-27 00:20:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.053934382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029390596s FAIL (2025-07-27 00:20:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ul 27 00:19:25.706249288 (2h27m41.487182654s) [seg_pos={0 0}, upload=2270]: Done begin upload segment. Jul 27 00:19:25.706293236 (2h27m41.487226619s) [noise=true , address=64.49.13.140:4514 , piece_id=H5XDPXAAXSOADAMLEDVSZM7ZWNAGZ3UUCOJBPPY2HEZJM73O3YRA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.706312245 (2h27m41.487245629s) [noise=true , address=104.36.82.53:4149 , piece_id=4Z3PWW6ORAURBN2QXMP3X2PHISDIOLBEPTX4PYF5HJFIGOSSP6EA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.706775540 (2h27m41.487708927s) [noise=true , address=79.127.205.246:4258 , piece_id=NIB3JHTSBTGJCFIR66BEQ5EQVMG6WOUQOQFYGDLZHBA7RXUNJGRA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.706911155 (2h27m41.487844541s) [noise=true , address=77.90.31.12:4615 , piece_id=THUY33GOVCFFMPLRYB73VSMWMV5XG7PV3GREJ2S2QYYBII2QZHJQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707336964 (2h27m41.488270349s) [noise=true , address=79.127.205.245:4242 , piece_id=Z5YHTFBGQ6N4NBKQ4GGLGXXROTANZIWRPQXJATSRXU6QDRRWRLRA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707404411 (2h27m41.488337797s) [noise=true , address=156.146.43.225:4448 , piece_id=L3UBVGMRBW2LJ4IEKFDIW2BQQSWRTAESUI4VELS4DQH4D4ZC65MQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707758778 (2h27m41.488692166s) [noise=true , address=104.36.82.39:4755 , piece_id=S2CUQL25DE7CPJC3N5YV3O5SOA6SRZI6J4LKJCG4PDJP2TX3HNWA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.707854359 (2h27m41.488787749s) [noise=true , address=79.127.219.38:4010 , piece_id=3XWNV2ZLKPEJBWSAFU7MAYKXPE36QQT2Q4TQXUZNGTD6FYR2MLMQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.708162682 (2h27m41.489096076s) [noise=true , address=79.127.205.234:4604 , piece_id=WY3NEWC46GRELOHU6CR2N6HQD4BXFLZVVNREZ452X7PAZFLDMVWA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.708544862 (2h27m41.489478248s) [noise=true , address=79.127.220.98:4422 , piece_id=WKVGGMAFY3WFBJ53ZXDWIAZB3MLBUQHVFU2CTI4R5YKU2ZSSL5SQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.708726535 (2h27m41.489659938s) [noise=true , address=104.36.82.11:4116 , piece_id=PSOPMK33F6S7FTCRVSSP2RFUFPPTH36Z5WWFXGUJQH5YAZYPARAQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.709121937 (2h27m41.490055324s) [noise=true , address=104.36.82.54:4189 , piece_id=IN6RS3SW7UVXELEO3WXD3ZKO7LZBAIYYSQ2HPADY3JMKKKDGJQVQ, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.711105701 (2h27m41.492039088s) [noise=true , address=77.90.31.19:4875 , piece_id=L6RKXJCWEUCYLD6277BBLIZPNKSFR7KHF6QPWHSDHVKNGX6543HA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.711232622 (2h27m41.492166005s) [noise=true , address=104.36.82.46:4512 , piece_id=CH3PXUGB6V53UYRLDXDYP3TXDG6BZP3NJEN3LHL6X37E5QRS4G3A, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.711871366 (2h27m41.492804755s) [noise=true , address=79.127.205.235:4681 , piece_id=YNMMADATC3QZA5FOYS3S3DYZRYVG7XR6N6NGJUD5WLDMGCIWWWEA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.713814558 (2h27m41.494747946s) [noise=true , address=77.90.31.10:4540 , piece_id=I255I52UZO2YFNHA23ADQK4LFGS3IONE2VVMC3H34T6VTNPEZKQA, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.714466206 (2h27m41.495399594s) [noise=true , address=104.36.82.47:4523 , piece_id=ODCOGUJWDQUYE6U7SHNYBBCHW7Z7HPLHAOVLBSNDWTAUML4STD2A, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.715059784 (2h27m41.495993171s) [noise=true , address=79.127.201.215:4334 , piece_id=TH6F2VG27ZCD3H3CNVAX2GJK2XASQWGR5KQLDLFSKCAROG6K6F7Q, seg_pos={0 0}, upload=2270]: Uploading piece... Jul 27 00:19:25.802847787 (2h27m41.583781173s) [upload=2270]: Next returned nil segment Jul 27 00:19:25.802885643 (2h27m41.583819037s) [upload=2270]: Waiting for error group managing segments... Jul 27 00:19:25.811137945 (2h27m41.592071307s) [noise=true , address=79.127.219.38:4010 , piece_id=3XWNV2ZLKPEJBWSAFU7MAYKXPE36QQT2Q4TQXUZNGTD6FYR2MLMQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:25.957278170 (2h27m41.738211546s) [noise=true , address=79.127.220.98:4422 , piece_id=WKVGGMAFY3WFBJ53ZXDWIAZB3MLBUQHVFU2CTI4R5YKU2ZSSL5SQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE, address:79.127.220.98:4422): protocol: storage node overloaded, request limit: 2000 Jul 27 00:19:26.063714809 (2h27m41.844648185s) [noise=true , address=79.127.201.215:4334 , piece_id=TH6F2VG27ZCD3H3CNVAX2GJK2XASQWGR5KQLDLFSKCAROG6K6F7Q, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.336237481 (2h27m42.11717086s) [noise=true , address=79.127.205.246:4258 , piece_id=NIB3JHTSBTGJCFIR66BEQ5EQVMG6WOUQOQFYGDLZHBA7RXUNJGRA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.347751756 (2h27m42.128685126s) [noise=true , address=79.127.205.234:4604 , piece_id=WY3NEWC46GRELOHU6CR2N6HQD4BXFLZVVNREZ452X7PAZFLDMVWA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.382836995 (2h27m42.163770363s) [noise=true , address=79.127.205.235:4681 , piece_id=YNMMADATC3QZA5FOYS3S3DYZRYVG7XR6N6NGJUD5WLDMGCIWWWEA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:26.831735028 (2h27m42.612668415s) [noise=true , address=79.127.205.245:4242 , piece_id=Z5YHTFBGQ6N4NBKQ4GGLGXXROTANZIWRPQXJATSRXU6QDRRWRLRA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:27.768486357 (2h27m43.549419723s) [noise=true , address=64.49.13.140:4514 , piece_id=H5XDPXAAXSOADAMLEDVSZM7ZWNAGZ3UUCOJBPPY2HEZJM73O3YRA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:29.992944619 (2h27m45.773877996s) [noise=true , address=156.146.43.225:4448 , piece_id=L3UBVGMRBW2LJ4IEKFDIW2BQQSWRTAESUI4VELS4DQH4D4ZC65MQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:34.937964336 (2h27m50.718897705s) [noise=true , address=77.90.31.12:4615 , piece_id=THUY33GOVCFFMPLRYB73VSMWMV5XG7PV3GREJ2S2QYYBII2QZHJQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:19:38.553526460 (2h27m54.334459829s) [noise=true , address=77.90.31.19:4875 , piece_id=L6RKXJCWEUCYLD6277BBLIZPNKSFR7KHF6QPWHSDHVKNGX6543HA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: <nil> Jul 27 00:20:25.585109159 (2h28m41.366042536s) [noise=true , address=104.36.82.46:4512 , piece_id=CH3PXUGB6V53UYRLDXDYP3TXDG6BZP3NJEN3LHL6X37E5QRS4G3A, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:25.585469865 (2h28m41.366403241s) [noise=true , address=104.36.82.39:4755 , piece_id=S2CUQL25DE7CPJC3N5YV3O5SOA6SRZI6J4LKJCG4PDJP2TX3HNWA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1cmNqubJ82X6bTLTn5zmMD7HhVDKyP8SH9vo2ZYq5S6GmX6Wpx, address:104.36.82.39:4755): protocol: write tcp 159.223.143.39:39766->104.36.82.39:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585534272 (2h28m41.366467639s) [noise=true , address=104.36.82.11:4116 , piece_id=PSOPMK33F6S7FTCRVSSP2RFUFPPTH36Z5WWFXGUJQH5YAZYPARAQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:123aoo9g11aJP9hiL8shu2dx44tjyKzEWqPUjHUdXtR7LLFyng8, address:104.36.82.11:4116): protocol: write tcp 159.223.143.39:39408->104.36.82.11:4116: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585574078 (2h28m41.366507462s) [noise=true , address=77.90.31.10:4540 , piece_id=I255I52UZO2YFNHA23ADQK4LFGS3IONE2VVMC3H34T6VTNPEZKQA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: write tcp 159.223.143.39:60792->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585615670 (2h28m41.366549032s) [noise=true , address=104.36.82.53:4149 , piece_id=4Z3PWW6ORAURBN2QXMP3X2PHISDIOLBEPTX4PYF5HJFIGOSSP6EA, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585631670 (2h28m41.366565051s) [noise=true , address=104.36.82.47:4523 , piece_id=ODCOGUJWDQUYE6U7SHNYBBCHW7Z7HPLHAOVLBSNDWTAUML4STD2A, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 159.223.143.39:40386->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:20:25.585787287 (2h28m41.366720667s) [noise=true , address=104.36.82.54:4189 , piece_id=IN6RS3SW7UVXELEO3WXD3ZKO7LZBAIYYSQ2HPADY3JMKKKDGJQVQ, seg_pos={0 0}, upload=2270]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:20:25.585835115 (2h28m41.366768492s) [seg_pos={0 0}, upload=2270]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:20:25.611663486 (2h28m41.392596855s) [upload=2270]: Done uploading segments... Jul 27 00:20:25.611688524 (2h28m41.392621893s) [upload=2270]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.029407984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.XXX...............X..X......X.XX....X...X...............] duration: 28.002856191s FAIL (2025-07-27 00:20:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.003134514s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X..............X.X........X...XX............................] duration: 46.693315509s FAIL (2025-07-27 00:19:49 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.693342694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....] duration: 557.523641ms FAIL (2025-07-27 00:19:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 557.55777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.028967626s FAIL (2025-07-27 00:19:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 13628s) [upload=2253]: Uploading segments... Jul 27 00:18:25.505447450 (2h26m41.286380826s) [upload=2253]: Got next segment. Inline: false Jul 27 00:18:25.505497438 (2h26m41.286430814s) [upload=2253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:18:25.505503257 (2h26m41.286436634s) [upload=2253]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:18:25.542470391 (2h26m41.323403766s) [seg_pos={0 0}, upload=2253]: Begin upload segment... Jul 27 00:18:25.542524862 (2h26m41.323458237s) [seg_pos={0 0}, upload=2253]: Done begin upload segment. Jul 27 00:18:25.542551327 (2h26m41.323484708s) [noise=true , address=79.127.205.234:4620 , piece_id=LLVX4NZ2Y72PBSEIHTB4FYEI3NWRVLG36CPAEGVILGQXUFIRZ3GA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.542574512 (2h26m41.323507887s) [noise=true , address=77.90.31.16:4755 , piece_id=UEBN6CJTXQUTGCRNLQTNVU7W6AEQWRWWEJYOM5EB5TQWETRUPYTA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.542978014 (2h26m41.323911399s) [noise=true , address=156.146.43.225:4448 , piece_id=6TCBZVCNR7UWKUYZQ32J7GPLXITM6XAQ2426ZIFLREB4D5RPPQFA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.542999510 (2h26m41.323932888s) [noise=true , address=79.127.201.215:4345 , piece_id=NOUYE6SVY5G7BX2N26TII4IP62B27SB2FAQ6VXMVCAJLJBTCU76A, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543392469 (2h26m41.324325855s) [noise=true , address=77.90.31.15:4717 , piece_id=Q4ATETGUL4XFWGLQNBYT7QBJGWSUCQSLU6NITNVHGQTF6JPVTCSQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543407344 (2h26m41.324340722s) [noise=true , address=79.127.220.101:4041 , piece_id=3XWLWEOYSODRVMXWZ4URMKIFDRCFOTDZB3WZJJIVIRHO42FAAZ2A, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543810392 (2h26m41.324743773s) [noise=true , address=77.90.31.21:4910 , piece_id=A663M76HRVEMVJ6KRUKEOFBOQ7PD44OU5CI7RFX5RUFKDOPCTBWQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.543856966 (2h26m41.32479035s) [noise=true , address=79.127.205.233:4650 , piece_id=VSOY4IGX4NDAKQ2ZEWZTGUFU65JIWXSZEV7EGCBKU7F6AVHZF3RA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544407014 (2h26m41.325340398s) [noise=true , address=64.49.13.138:4482 , piece_id=XNQYZQUTUY4FD26EP5DMDRZXH6UEHWEBIEJNT4QORQJK7TOKXSLQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544458776 (2h26m41.325392158s) [noise=true , address=64.49.13.140:4508 , piece_id=HS4TXGBDK32Q45LXM5N2CBKWFH2JWLZXP3KEZWEDKZHVYA6BNNJQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544775396 (2h26m41.325708775s) [noise=true , address=77.90.31.17:4787 , piece_id=PDAC4LUZWBJDHUBYT6JSU2NC7ZK3K3N2T6UBSCPPF2WTVJ2WYODQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.544895001 (2h26m41.325828392s) [noise=true , address=104.36.82.47:4524 , piece_id=S3FOVUTRU47GSK7W4JGGD7SE6CPSA27WMXC356T6G4WH3VYFS2DQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.545225065 (2h26m41.32615845s) [noise=true , address=104.36.82.46:4484 , piece_id=RCW4XEBEVFGHJKKOFDFSVV5P4FXESZVBK4V3N5L6GIJKT4HEZTQA, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.545264602 (2h26m41.326197988s) [noise=true , address=79.127.205.235:4692 , piece_id=MDJBEPZURM4QBD45YFBDAVGK22FBMHZF5IIN5CQ3S56COQR3WGPQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.546316170 (2h26m41.327249556s) [noise=true , address=79.127.205.246:4283 , piece_id=LVZ4FH2P5XR4INTRI5LGG4CW23NIOYQF4LJ7KBSMNSO2NJ7UIFBQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.546671841 (2h26m41.327605231s) [noise=true , address=104.36.82.53:4141 , piece_id=477OMFJON5AN4TJDGKLKUB25DBPBX2ZCEXNIMTGEOMBYAP7DASUQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.546891803 (2h26m41.327825188s) [noise=true , address=79.127.205.245:4247 , piece_id=KUO6XSRZGHP4YHNB5S35BMC4GIZ5PI3P2LLZULM52EPCCLUM6JEQ, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.547610759 (2h26m41.328544141s) [noise=true , address=104.36.82.56:4257 , piece_id=ZSLRRQBIZ3R3FPSZTCKSWJ5CQI7FRZMDFKTVBN5W4EA5DEWQVL2Q, seg_pos={0 0}, upload=2253]: Uploading piece... Jul 27 00:18:25.631350909 (2h26m41.412284293s) [upload=2253]: Next returned nil segment Jul 27 00:18:25.631373306 (2h26m41.412306694s) [upload=2253]: Waiting for error group managing segments... Jul 27 00:18:25.677379328 (2h26m41.458312697s) [noise=true , address=77.90.31.15:4717 , piece_id=Q4ATETGUL4XFWGLQNBYT7QBJGWSUCQSLU6NITNVHGQTF6JPVTCSQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:25.679049752 (2h26m41.459983128s) [noise=true , address=77.90.31.16:4755 , piece_id=UEBN6CJTXQUTGCRNLQTNVU7W6AEQWRWWEJYOM5EB5TQWETRUPYTA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.129934537 (2h26m41.910867903s) [noise=true , address=79.127.205.233:4650 , piece_id=VSOY4IGX4NDAKQ2ZEWZTGUFU65JIWXSZEV7EGCBKU7F6AVHZF3RA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.205940754 (2h26m41.986874118s) [noise=true , address=79.127.205.234:4620 , piece_id=LLVX4NZ2Y72PBSEIHTB4FYEI3NWRVLG36CPAEGVILGQXUFIRZ3GA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.206893674 (2h26m41.987827037s) [noise=true , address=79.127.205.235:4692 , piece_id=MDJBEPZURM4QBD45YFBDAVGK22FBMHZF5IIN5CQ3S56COQR3WGPQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.207737374 (2h26m41.988670752s) [noise=true , address=79.127.205.246:4283 , piece_id=LVZ4FH2P5XR4INTRI5LGG4CW23NIOYQF4LJ7KBSMNSO2NJ7UIFBQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.230425581 (2h26m42.011358942s) [noise=true , address=79.127.205.245:4247 , piece_id=KUO6XSRZGHP4YHNB5S35BMC4GIZ5PI3P2LLZULM52EPCCLUM6JEQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:26.231838104 (2h26m42.01277148s) [noise=true , address=64.49.13.140:4508 , piece_id=HS4TXGBDK32Q45LXM5N2CBKWFH2JWLZXP3KEZWEDKZHVYA6BNNJQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:28.066848111 (2h26m43.847781489s) [noise=true , address=156.146.43.225:4448 , piece_id=6TCBZVCNR7UWKUYZQ32J7GPLXITM6XAQ2426ZIFLREB4D5RPPQFA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:29.603624553 (2h26m45.384557923s) [noise=true , address=79.127.220.101:4041 , piece_id=3XWLWEOYSODRVMXWZ4URMKIFDRCFOTDZB3WZJJIVIRHO42FAAZ2A, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:31.883139666 (2h26m47.664073043s) [noise=true , address=64.49.13.138:4482 , piece_id=XNQYZQUTUY4FD26EP5DMDRZXH6UEHWEBIEJNT4QORQJK7TOKXSLQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:18:41.511553208 (2h26m57.292486586s) [noise=true , address=79.127.201.215:4345 , piece_id=NOUYE6SVY5G7BX2N26TII4IP62B27SB2FAQ6VXMVCAJLJBTCU76A, seg_pos={0 0}, upload=2253]: Done uploading piece. err: <nil> Jul 27 00:19:25.475283152 (2h27m41.256216537s) [noise=true , address=77.90.31.21:4910 , piece_id=A663M76HRVEMVJ6KRUKEOFBOQ7PD44OU5CI7RFX5RUFKDOPCTBWQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: rpc: dial tcp 77.90.31.21:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475417500 (2h27m41.256350878s) [noise=true , address=104.36.82.53:4141 , piece_id=477OMFJON5AN4TJDGKLKUB25DBPBX2ZCEXNIMTGEOMBYAP7DASUQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475533294 (2h27m41.256466674s) [noise=true , address=77.90.31.17:4787 , piece_id=PDAC4LUZWBJDHUBYT6JSU2NC7ZK3K3N2T6UBSCPPF2WTVJ2WYODQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475553828 (2h27m41.256487198s) [noise=true , address=104.36.82.56:4257 , piece_id=ZSLRRQBIZ3R3FPSZTCKSWJ5CQI7FRZMDFKTVBN5W4EA5DEWQVL2Q, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: rpc: dial tcp 104.36.82.56:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475615507 (2h27m41.256548877s) [noise=true , address=104.36.82.46:4484 , piece_id=RCW4XEBEVFGHJKKOFDFSVV5P4FXESZVBK4V3N5L6GIJKT4HEZTQA, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:19:25.475646375 (2h27m41.256579739s) [noise=true , address=104.36.82.47:4524 , piece_id=S3FOVUTRU47GSK7W4JGGD7SE6CPSA27WMXC356T6G4WH3VYFS2DQ, seg_pos={0 0}, upload=2253]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: rpc: dial tcp 104.36.82.47:4524: i/o timeout; context deadline exceeded Jul 27 00:19:25.475685142 (2h27m41.256618528s) [seg_pos={0 0}, upload=2253]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:19:25.499876587 (2h27m41.280809969s) [upload=2253]: Done uploading segments... Jul 27 00:19:25.499902437 (2h27m41.280835808s) [upload=2253]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.02898918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XX..............X.X........X...XX...........................] duration: 45.150676992s FAIL (2025-07-27 00:19:02 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.150710876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..] duration: 2.033144609s FAIL (2025-07-27 00:19:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.033299529s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X....] duration: 1.97645334s FAIL (2025-07-27 00:18:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.976479369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.XXX...............X..X......X.XX....X...X..............] duration: 2.046254052s FAIL (2025-07-27 00:18:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.046358918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X...] duration: 2.047593574s FAIL (2025-07-27 00:18:34 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.047648109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030855327s FAIL (2025-07-27 00:18:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 3ZBYYPWQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.433429709 (2h25m41.21436309s) [noise=true , address=104.36.82.53:4121 , piece_id=5NX3GTJG4GVSAD3ELLCHAF2O2AKIZS5OLCYWFI2WXAYYEYXQB4VQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.433808462 (2h25m41.214741853s) [noise=true , address=79.127.219.47:4048 , piece_id=3IRM2Z4VJDDUUUNS55IEJ6QZDL3I7Z2VVXNME7KTK7BJ7KX2ZPAA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.434201983 (2h25m41.215135367s) [noise=true , address=156.146.43.225:4448 , piece_id=FOD34AK2IRA54YXSIQ677YDARBMQZEVXCVYXV4O54VRTLSXEGXQQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.434547297 (2h25m41.21548068s) [noise=true , address=79.127.205.235:4669 , piece_id=OYG3OI5WNHYTGHNXRC5XWAZ7K5HXMMPXHE2FCG5WJEYBESEPHP3Q, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.434944479 (2h25m41.215877905s) [noise=true , address=104.36.82.54:4169 , piece_id=P5K5LAJSGGIG45SCX5P74A7ZH6BBKPRO26TFVF7I523LOWWOCSRQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435089878 (2h25m41.216023258s) [noise=true , address=104.36.82.47:4540 , piece_id=IKVWDCQM7PCG3BK4IFGLEZTHVUTFNOLYGWJCONO7OJGO27KXGWGQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435415377 (2h25m41.216348765s) [noise=true , address=79.127.201.214:4289 , piece_id=3ZTNGA4JFHXXPA4SNPSDTSE4A7BO2XEDYDPDC3FYH3DPGBT5RDYA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435809500 (2h25m41.21674289s) [noise=true , address=79.127.205.246:4280 , piece_id=U2NWGWBUZ56YBIX6JCS4WWAWPJKQL4H6U4OEJLMRGLYMXRPFCDLA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.435832958 (2h25m41.216766344s) [noise=true , address=104.36.82.56:4252 , piece_id=4C2CKS62DVOK4FD7ANAYPFSWA44RPIFCPAQ2JMSKGBCQVWBDT4BA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.436290772 (2h25m41.217224135s) [noise=true , address=79.127.220.101:4041 , piece_id=V2HCCLSBVIJ3EHKLGZQTUG2NBKLGXBTLWASNU2YTAUFJLMEJTC2Q, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.436645989 (2h25m41.217579376s) [noise=true , address=104.36.82.46:4486 , piece_id=IRF7MBR3UJ2I66DUAE7WRDYIXTT4KN3B744BR3ZSD5M6KMNX2YGQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.437068321 (2h25m41.218001713s) [noise=true , address=64.49.13.154:4927 , piece_id=4EU77ZGSLMYQL3IW4EITP6HNOKCTV34UE6IXYZQAAGLGGGTSYMHA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.437470784 (2h25m41.218404164s) [noise=true , address=45.91.116.14:4206 , piece_id=SJ242HH354E2I64RMLODFM5ILO4JEIQJ533LESMFDMO43QC6SYZQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.437806428 (2h25m41.218739811s) [noise=true , address=79.127.201.215:4360 , piece_id=LPDOKJ5RTBNZN6RK2CKYSDWW3V2ZZSSLBS45IMJW63LAFPOIP2DQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.438208746 (2h25m41.219142133s) [noise=true , address=77.90.31.15:4707 , piece_id=6MUCOOI2GWNTJWKMBCMEGGOPABZGTNMJCBRJINLI2DBNH5MDTSRA, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.438628794 (2h25m41.219562176s) [noise=true , address=104.36.82.52:4110 , piece_id=2L3FU2G2XJ7KXWKZQFMMFKNUTGFDDIVX6WZ7JDA7ZGR27TUY5XFQ, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.444249530 (2h25m41.22518291s) [noise=true , address=79.127.205.233:4633 , piece_id=5ZH4CAIECCGSIUHN6SUURPMJDBZJHXZEM5Y6TUCQ6CGLFVLIGQ5A, seg_pos={0 0}, upload=2236]: Uploading piece... Jul 27 00:17:25.512169007 (2h25m41.293102371s) [upload=2236]: Next returned nil segment Jul 27 00:17:25.512193731 (2h25m41.293127093s) [upload=2236]: Waiting for error group managing segments... Jul 27 00:17:25.538005020 (2h25m41.318938397s) [noise=true , address=77.90.31.15:4707 , piece_id=6MUCOOI2GWNTJWKMBCMEGGOPABZGTNMJCBRJINLI2DBNH5MDTSRA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:25.565397624 (2h25m41.34633101s) [noise=true , address=79.127.220.101:4041 , piece_id=V2HCCLSBVIJ3EHKLGZQTUG2NBKLGXBTLWASNU2YTAUFJLMEJTC2Q, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:25.576511888 (2h25m41.357445269s) [noise=true , address=156.146.43.225:4448 , piece_id=FOD34AK2IRA54YXSIQ677YDARBMQZEVXCVYXV4O54VRTLSXEGXQQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:25.587314374 (2h25m41.368247752s) [noise=true , address=79.127.201.214:4289 , piece_id=3ZTNGA4JFHXXPA4SNPSDTSE4A7BO2XEDYDPDC3FYH3DPGBT5RDYA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:26.035522735 (2h25m41.816456143s) [noise=true , address=79.127.205.235:4669 , piece_id=OYG3OI5WNHYTGHNXRC5XWAZ7K5HXMMPXHE2FCG5WJEYBESEPHP3Q, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:26.171465350 (2h25m41.952398727s) [noise=true , address=79.127.205.246:4280 , piece_id=U2NWGWBUZ56YBIX6JCS4WWAWPJKQL4H6U4OEJLMRGLYMXRPFCDLA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:26.353367125 (2h25m42.134300514s) [noise=true , address=79.127.205.233:4633 , piece_id=5ZH4CAIECCGSIUHN6SUURPMJDBZJHXZEM5Y6TUCQ6CGLFVLIGQ5A, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:28.875651463 (2h25m44.656584832s) [noise=true , address=79.127.219.47:4048 , piece_id=3IRM2Z4VJDDUUUNS55IEJ6QZDL3I7Z2VVXNME7KTK7BJ7KX2ZPAA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:34.668860712 (2h25m50.449794094s) [noise=true , address=45.91.116.14:4206 , piece_id=SJ242HH354E2I64RMLODFM5ILO4JEIQJ533LESMFDMO43QC6SYZQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 27 00:17:35.746949652 (2h25m51.527883016s) [noise=true , address=45.91.116.10:4093 , piece_id=LRNUFEZLW3THSKOTCIRUM2KMAYOEFCZWQG5HIMRAN7LQ3ZBYYPWQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:36.597549852 (2h25m52.378483231s) [noise=true , address=64.49.13.154:4927 , piece_id=4EU77ZGSLMYQL3IW4EITP6HNOKCTV34UE6IXYZQAAGLGGGTSYMHA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:17:58.869242533 (2h26m14.6501759s) [noise=true , address=79.127.201.215:4360 , piece_id=LPDOKJ5RTBNZN6RK2CKYSDWW3V2ZZSSLBS45IMJW63LAFPOIP2DQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: <nil> Jul 27 00:18:25.363436895 (2h26m41.144370271s) [noise=true , address=104.36.82.56:4252 , piece_id=4C2CKS62DVOK4FD7ANAYPFSWA44RPIFCPAQ2JMSKGBCQVWBDT4BA, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1UDFNiqXb4riRuewJDPrKFykVWLiryNKqgkH6uCZszz6Sr8naB, address:104.36.82.56:4252): protocol: rpc: dial tcp 104.36.82.56:4252: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363499012 (2h26m41.144432382s) [noise=true , address=104.36.82.52:4110 , piece_id=2L3FU2G2XJ7KXWKZQFMMFKNUTGFDDIVX6WZ7JDA7ZGR27TUY5XFQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363574402 (2h26m41.144507767s) [noise=true , address=104.36.82.53:4121 , piece_id=5NX3GTJG4GVSAD3ELLCHAF2O2AKIZS5OLCYWFI2WXAYYEYXQB4VQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: write tcp 159.223.143.39:56734->104.36.82.53:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363634515 (2h26m41.144567878s) [noise=true , address=104.36.82.47:4540 , piece_id=IKVWDCQM7PCG3BK4IFGLEZTHVUTFNOLYGWJCONO7OJGO27KXGWGQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:104.36.82.47:4540): protocol: rpc: dial tcp 104.36.82.47:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363660665 (2h26m41.144594041s) [noise=true , address=104.36.82.46:4486 , piece_id=IRF7MBR3UJ2I66DUAE7WRDYIXTT4KN3B744BR3ZSD5M6KMNX2YGQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic, address:104.36.82.46:4486): protocol: rpc: dial tcp 104.36.82.46:4486: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363722746 (2h26m41.144656122s) [noise=true , address=104.36.82.54:4169 , piece_id=P5K5LAJSGGIG45SCX5P74A7ZH6BBKPRO26TFVF7I523LOWWOCSRQ, seg_pos={0 0}, upload=2236]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:25.363756447 (2h26m41.144689839s) [seg_pos={0 0}, upload=2236]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:18:25.391608850 (2h26m41.172542217s) [upload=2236]: Done uploading segments... Jul 27 00:18:25.391646216 (2h26m41.172579581s) [upload=2236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.030876395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X.] duration: 1.94573129s FAIL (2025-07-27 00:18:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.9457545s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X] duration: 30.027866191s FAIL (2025-07-27 00:18:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027893431s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.XXX...............X..X......X.XX....X...X.............] duration: 1.982682937s FAIL (2025-07-27 00:18:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.982708187s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.XX..............X.X........X...XX..........................] duration: 37.322683533s FAIL (2025-07-27 00:17:46 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.322715592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031465588s FAIL (2025-07-27 00:17:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m41.066136758s) [upload=2221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:16:25.285209159 (2h24m41.066142533s) [upload=2221]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:16:25.319767806 (2h24m41.100701192s) [seg_pos={0 0}, upload=2221]: Begin upload segment... Jul 27 00:16:25.319831084 (2h24m41.100764457s) [seg_pos={0 0}, upload=2221]: Done begin upload segment. Jul 27 00:16:25.319865681 (2h24m41.100799063s) [noise=true , address=104.36.82.47:4525 , piece_id=SYVPP53DTYQ5K45AZKJZTA3BIB5ANJWLCZYHDZ3JBCLXPBREHMQQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.320389233 (2h24m41.101322621s) [noise=true , address=79.127.205.235:4685 , piece_id=J2NJBTM7GUEKBWQOBIWCR3V5AU6Q32VFZJTMBN2LOLRYEQFVSRBQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.320626047 (2h24m41.101559427s) [noise=true , address=64.49.13.154:4927 , piece_id=UUTIFM5HZIJCFUXN6SL4KR2ZQWZMSTDAZXD4R6E4IYDMA2TM2OIA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.320981090 (2h24m41.10191447s) [noise=true , address=104.36.82.5:4870 , piece_id=GPGSASZ3DHWYK3TFROF7YPESLPCCEVOQCCJXCR62VPUIELHNGHLQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.321066740 (2h24m41.102000126s) [noise=true , address=64.49.13.138:4482 , piece_id=P7ICQS4NSIEDV7DDVMAT2MQTMDV6HL3D4O4UUJR4NZGP6QB73OKA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.321572572 (2h24m41.102505959s) [noise=true , address=156.146.43.225:4434 , piece_id=4OWXBZ2WOK44KLD7QSSEPPA2AS3HUULUPTRTRPZ5636GXEAFTBFA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.321764012 (2h24m41.102697399s) [noise=true , address=79.127.220.98:4422 , piece_id=5MTPUQJFF2PT76S4QIAOACBARGZWVWV4RNBZR7I6SRDH5BQW66JA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322124162 (2h24m41.103057542s) [noise=true , address=79.127.205.234:4618 , piece_id=RXXCZKZBRZTT7M5BJA3D7RYOOPZBY6CPE2SL3P22MBX5RGMW6ALA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322142587 (2h24m41.103075972s) [noise=true , address=64.49.13.139:4478 , piece_id=V2VZ7OBAWPRTL4BRGHVKMLQ6LNMKGGAFJLY42ZPP5LNHS2MILFYA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322451838 (2h24m41.103385228s) [noise=true , address=79.127.219.47:4048 , piece_id=T63KXMHJDL5YD6QMNEKX5SGJRGT6ET7FNH67ZJ4RJ55IN6PPIHIA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.322583430 (2h24m41.103516807s) [noise=true , address=79.127.205.233:4623 , piece_id=5RVMBBL6M77NOHVCDF3BQB7MJXF23QDWS2C4GI6E23CTXLWF77AA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323110073 (2h24m41.104043455s) [noise=true , address=79.127.201.215:4343 , piece_id=7NDNNBLR5GHIFYYGJCDWFTGB2FPUOOUBIPQO5UO7GEHKMXSCQW6A, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323166956 (2h24m41.104100336s) [noise=true , address=104.36.82.46:4506 , piece_id=KY63MZ3U5CHALKJOSYHEXG3SAHMG4Z5WHZ5IYJZXG54UALSF77NQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323565464 (2h24m41.104498848s) [noise=true , address=79.127.205.245:4225 , piece_id=IFZEESA7RZJPNFKMEJXHLUPQNBTRTBZC4RF2HC7QPGR5C54GCJ6Q, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323762115 (2h24m41.104695494s) [noise=true , address=45.91.116.14:4206 , piece_id=CUSXUDHNMSNVHDS7GZETEFYQOUHEGEOUZQXADSZRNPKE4Z6AJMJA, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.323889183 (2h24m41.104822563s) [noise=true , address=104.36.82.55:4215 , piece_id=ZEDZFH2P2R2UUSNOSUK3N2W6WIWX64KKY3YOCS6GJW5YDZ3K44EQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.324445314 (2h24m41.105378702s) [noise=true , address=77.90.31.18:4833 , piece_id=QX7KRNGVZDHHQ2WU3RBFRZLRI3LAC4HFV75MDNNWNFZT2344J25A, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.324637924 (2h24m41.10557131s) [noise=true , address=104.36.82.54:4176 , piece_id=RUB2MGBLQUVJFR3YAU3BTFHJXT5SITU5GYXD4RSQF5KTAVP3HXKQ, seg_pos={0 0}, upload=2221]: Uploading piece... Jul 27 00:16:25.405885455 (2h24m41.186818824s) [upload=2221]: Next returned nil segment Jul 27 00:16:25.405915264 (2h24m41.186848646s) [upload=2221]: Waiting for error group managing segments... Jul 27 00:16:25.738834051 (2h24m41.519767419s) [noise=true , address=156.146.43.225:4434 , piece_id=4OWXBZ2WOK44KLD7QSSEPPA2AS3HUULUPTRTRPZ5636GXEAFTBFA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.925164489 (2h24m41.70609785s) [noise=true , address=79.127.205.235:4685 , piece_id=J2NJBTM7GUEKBWQOBIWCR3V5AU6Q32VFZJTMBN2LOLRYEQFVSRBQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.925277473 (2h24m41.706210837s) [noise=true , address=64.49.13.138:4482 , piece_id=P7ICQS4NSIEDV7DDVMAT2MQTMDV6HL3D4O4UUJR4NZGP6QB73OKA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 27 00:16:25.932252393 (2h24m41.713185754s) [noise=true , address=79.127.205.234:4618 , piece_id=RXXCZKZBRZTT7M5BJA3D7RYOOPZBY6CPE2SL3P22MBX5RGMW6ALA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.936905485 (2h24m41.71783885s) [noise=true , address=79.127.205.233:4623 , piece_id=5RVMBBL6M77NOHVCDF3BQB7MJXF23QDWS2C4GI6E23CTXLWF77AA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:25.937811783 (2h24m41.718745151s) [noise=true , address=79.127.205.245:4225 , piece_id=IFZEESA7RZJPNFKMEJXHLUPQNBTRTBZC4RF2HC7QPGR5C54GCJ6Q, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:26.249105509 (2h24m42.03003888s) [noise=true , address=45.91.116.14:4206 , piece_id=CUSXUDHNMSNVHDS7GZETEFYQOUHEGEOUZQXADSZRNPKE4Z6AJMJA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:27.056267585 (2h24m42.837200967s) [noise=true , address=64.49.13.139:4478 , piece_id=V2VZ7OBAWPRTL4BRGHVKMLQ6LNMKGGAFJLY42ZPP5LNHS2MILFYA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 27 00:16:27.847906157 (2h24m43.628839525s) [noise=true , address=79.127.219.47:4048 , piece_id=T63KXMHJDL5YD6QMNEKX5SGJRGT6ET7FNH67ZJ4RJ55IN6PPIHIA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:32.415161600 (2h24m48.196094969s) [noise=true , address=77.90.31.18:4833 , piece_id=QX7KRNGVZDHHQ2WU3RBFRZLRI3LAC4HFV75MDNNWNFZT2344J25A, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:33.390197542 (2h24m49.171130906s) [noise=true , address=79.127.220.98:4422 , piece_id=5MTPUQJFF2PT76S4QIAOACBARGZWVWV4RNBZR7I6SRDH5BQW66JA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:35.324489552 (2h24m51.105422921s) [noise=true , address=64.49.13.154:4927 , piece_id=UUTIFM5HZIJCFUXN6SL4KR2ZQWZMSTDAZXD4R6E4IYDMA2TM2OIA, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:16:38.959115923 (2h24m54.740049292s) [noise=true , address=79.127.201.215:4343 , piece_id=7NDNNBLR5GHIFYYGJCDWFTGB2FPUOOUBIPQO5UO7GEHKMXSCQW6A, seg_pos={0 0}, upload=2221]: Done uploading piece. err: <nil> Jul 27 00:17:25.256581505 (2h25m41.03751488s) [noise=true , address=104.36.82.46:4506 , piece_id=KY63MZ3U5CHALKJOSYHEXG3SAHMG4Z5WHZ5IYJZXG54UALSF77NQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: rpc: dial tcp 104.36.82.46:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256643185 (2h25m41.037576566s) [noise=true , address=104.36.82.54:4176 , piece_id=RUB2MGBLQUVJFR3YAU3BTFHJXT5SITU5GYXD4RSQF5KTAVP3HXKQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256722308 (2h25m41.037655674s) [noise=true , address=104.36.82.47:4525 , piece_id=SYVPP53DTYQ5K45AZKJZTA3BIB5ANJWLCZYHDZ3JBCLXPBREHMQQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: rpc: dial tcp 104.36.82.47:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256804080 (2h25m41.037737459s) [noise=true , address=104.36.82.5:4870 , piece_id=GPGSASZ3DHWYK3TFROF7YPESLPCCEVOQCCJXCR62VPUIELHNGHLQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:1MmdhnDp6kKktFb5btZUDeVQ2u3CNUKCZU7RfuEqiSQ2TEqwKX, address:104.36.82.5:4870): protocol: rpc: dial tcp 104.36.82.5:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256876115 (2h25m41.037809494s) [noise=true , address=104.36.82.55:4215 , piece_id=ZEDZFH2P2R2UUSNOSUK3N2W6WIWX64KKY3YOCS6GJW5YDZ3K44EQ, seg_pos={0 0}, upload=2221]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:49220->104.36.82.55:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:25.256931590 (2h25m41.037864972s) [seg_pos={0 0}, upload=2221]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:17:25.285948243 (2h25m41.06688162s) [upload=2221]: Done uploading segments... Jul 27 00:17:25.285977783 (2h25m41.066911162s) [upload=2221]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.031482318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.XXX...............X..X......X.XX....X...X............] duration: 1.841856237s FAIL (2025-07-27 00:17:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.841889508s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X..] duration: 1.213587711s FAIL (2025-07-27 00:17:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.213630039s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....X.XXX...............X..X......X.XX....X...X...........] duration: 1.92336075s FAIL (2025-07-27 00:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.923382281s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.] duration: 536.469461ms FAIL (2025-07-27 00:16:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.501926ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..XX..............X.X........X...XX.........................] duration: 13.239966644s FAIL (2025-07-27 00:16:36 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.239995807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027783341s FAIL (2025-07-27 00:16:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 877697s) [upload=2206]: Got next segment. Inline: false Jul 27 00:15:25.204124474 (2h23m40.985057875s) [upload=2206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:25.204133499 (2h23m40.985066888s) [upload=2206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:25.246103485 (2h23m41.027036869s) [seg_pos={0 0}, upload=2206]: Begin upload segment... Jul 27 00:15:25.246165900 (2h23m41.027099266s) [seg_pos={0 0}, upload=2206]: Done begin upload segment. Jul 27 00:15:25.246193432 (2h23m41.02712681s) [noise=true , address=79.127.205.245:4240 , piece_id=RTEASNNBTDSX7XTLRQZJOYHCLI47SUQBKVABIXC5BHVYIGFGZ63A, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.246622239 (2h23m41.027555625s) [noise=true , address=79.127.205.235:4690 , piece_id=WJI6X4YMEM3RIYSNFZAAGRLP2BQNO3Z37GPM7E4VBKNRLAK7LUWQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.246962606 (2h23m41.027895986s) [noise=true , address=104.36.82.54:4166 , piece_id=P7ZZSYJMEKFEEPXPNPMB5PYOWMRVNNUH3KEFH5F7CNGZ4QMH3KOQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247172901 (2h23m41.028106282s) [noise=true , address=79.127.219.38:4001 , piece_id=JFTNZ727CPV66POZJXI4NK7TBDZG3OR3YXTVT3SGO3JZV7VKU2NQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247301713 (2h23m41.028235094s) [noise=true , address=77.90.31.15:4707 , piece_id=MDPCCZJW7EZZDJ7YWXD27F3H57JJJ3275HZ4EUWGOWOBPCNIYKNQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247616338 (2h23m41.02854974s) [noise=true , address=79.127.201.215:4359 , piece_id=SBYNPEMOYTGTAVUO6WILDNJRDXRKIV6PNM52NEUPUDZS3GQAC4LQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.247662824 (2h23m41.028596202s) [noise=true , address=77.90.31.10:4531 , piece_id=ZBSKCZZP6GCYJSKC5FUN5SYZSGC4JSHJBP6U2FAOBJOKIV2Z4UNQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.248173859 (2h23m41.02910724s) [noise=true , address=79.127.220.97:4361 , piece_id=SKOHDTJH63XK3VQSTFOYKE7FUN6SK76KRPL4ZJJ7VTX55DNN5TNQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.248568305 (2h23m41.029501686s) [noise=true , address=156.146.43.225:4434 , piece_id=R2OKXNV47534RXMOE5YDLA3OBIK5DIHWYILJDQWSURZLPGZFQ3ZQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.248988972 (2h23m41.029922352s) [noise=true , address=77.90.31.12:4616 , piece_id=YO7377KIVG6GKATASD7NWZVY7NA2AJPQE6KD2VTXQ46MVXVDYQ7A, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.249443834 (2h23m41.030377212s) [noise=true , address=104.36.82.47:4545 , piece_id=RP6FNM4QTH5Z5MJO2FDPWLI7KLKOLV2WXTIM4MM2JFWDFSN7XCCA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.249851997 (2h23m41.030785381s) [noise=true , address=64.49.13.140:4508 , piece_id=LVHMPRDJNA6TXGOJSNO7YQIJU2J4U3E2OANN7JUYE3GQC6Y23BIQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.250276585 (2h23m41.031209967s) [noise=true , address=104.36.82.53:4128 , piece_id=XL6U67NQKSCBYJ3M5VC5XZ5CT43LTLILRZ2XQQLNUXKAI2QKD2JA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.250717679 (2h23m41.031651062s) [noise=true , address=79.127.201.214:4321 , piece_id=BLSZTKBBI6EPQZJBTKCYA2Q2NFBOVIMUHMFZYGBIG2ELZ2W5RERA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.251568787 (2h23m41.032502169s) [noise=true , address=64.49.13.154:4927 , piece_id=DPT2QCUKTUW7HUQCZTONGC4AZVPR54UROP4PHZBBQS2WJCQJUTXA, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.252833752 (2h23m41.033767134s) [noise=true , address=79.127.220.101:4041 , piece_id=C4U4ID7UKFLZSZUTCLLKEDDV63BZOXB5AAPWXVTNGOIEWS3EMNLQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.253154113 (2h23m41.034087491s) [noise=true , address=79.127.205.234:4618 , piece_id=EIX3FPJ3PO6EAAEFYYSVQFB5E3G2YUFGFP6XZ7GEGWHSVWKL3O2Q, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.253411742 (2h23m41.03434512s) [noise=true , address=77.90.31.21:4896 , piece_id=CRMTZHBF7D7S46KFGNDS676EAP3EOVF5NUH6RTYFUY4C62TKM2UQ, seg_pos={0 0}, upload=2206]: Uploading piece... Jul 27 00:15:25.317217016 (2h23m41.098150412s) [upload=2206]: Next returned nil segment Jul 27 00:15:25.317238420 (2h23m41.098171804s) [upload=2206]: Waiting for error group managing segments... Jul 27 00:15:25.323642403 (2h23m41.104575783s) [noise=true , address=79.127.219.38:4001 , piece_id=JFTNZ727CPV66POZJXI4NK7TBDZG3OR3YXTVT3SGO3JZV7VKU2NQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.352837629 (2h23m41.133770999s) [noise=true , address=77.90.31.15:4707 , piece_id=MDPCCZJW7EZZDJ7YWXD27F3H57JJJ3275HZ4EUWGOWOBPCNIYKNQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.833445711 (2h23m41.614379088s) [noise=true , address=79.127.205.245:4240 , piece_id=RTEASNNBTDSX7XTLRQZJOYHCLI47SUQBKVABIXC5BHVYIGFGZ63A, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.903497841 (2h23m41.68443121s) [noise=true , address=79.127.205.235:4690 , piece_id=WJI6X4YMEM3RIYSNFZAAGRLP2BQNO3Z37GPM7E4VBKNRLAK7LUWQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:25.979282690 (2h23m41.760216077s) [noise=true , address=79.127.205.234:4618 , piece_id=EIX3FPJ3PO6EAAEFYYSVQFB5E3G2YUFGFP6XZ7GEGWHSVWKL3O2Q, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:26.176335879 (2h23m41.957269258s) [noise=true , address=64.49.13.154:4927 , piece_id=DPT2QCUKTUW7HUQCZTONGC4AZVPR54UROP4PHZBBQS2WJCQJUTXA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:26.266294508 (2h23m42.047227889s) [noise=true , address=79.127.220.97:4361 , piece_id=SKOHDTJH63XK3VQSTFOYKE7FUN6SK76KRPL4ZJJ7VTX55DNN5TNQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:26.353497230 (2h23m42.13443061s) [noise=true , address=64.49.13.140:4508 , piece_id=LVHMPRDJNA6TXGOJSNO7YQIJU2J4U3E2OANN7JUYE3GQC6Y23BIQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:15:29.161225666 (2h23m44.942159057s) [noise=true , address=79.127.201.214:4321 , piece_id=BLSZTKBBI6EPQZJBTKCYA2Q2NFBOVIMUHMFZYGBIG2ELZ2W5RERA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:29.667738457 (2h23m45.44867183s) [noise=true , address=79.127.220.101:4041 , piece_id=C4U4ID7UKFLZSZUTCLLKEDDV63BZOXB5AAPWXVTNGOIEWS3EMNLQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:30.515418509 (2h23m46.29635188s) [noise=true , address=156.146.43.225:4434 , piece_id=R2OKXNV47534RXMOE5YDLA3OBIK5DIHWYILJDQWSURZLPGZFQ3ZQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:45.149308791 (2h24m0.93024217s) [noise=true , address=77.90.31.12:4616 , piece_id=YO7377KIVG6GKATASD7NWZVY7NA2AJPQE6KD2VTXQ46MVXVDYQ7A, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:15:45.332301191 (2h24m1.113234552s) [noise=true , address=79.127.201.215:4359 , piece_id=SBYNPEMOYTGTAVUO6WILDNJRDXRKIV6PNM52NEUPUDZS3GQAC4LQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: <nil> Jul 27 00:16:25.172775922 (2h24m40.953709307s) [noise=true , address=104.36.82.53:4128 , piece_id=XL6U67NQKSCBYJ3M5VC5XZ5CT43LTLILRZ2XQQLNUXKAI2QKD2JA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 159.223.143.39:34196->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.172935510 (2h24m40.953868887s) [noise=true , address=104.36.82.47:4545 , piece_id=RP6FNM4QTH5Z5MJO2FDPWLI7KLKOLV2WXTIM4MM2JFWDFSN7XCCA, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173018375 (2h24m40.953951752s) [noise=true , address=77.90.31.21:4896 , piece_id=CRMTZHBF7D7S46KFGNDS676EAP3EOVF5NUH6RTYFUY4C62TKM2UQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: rpc: dial tcp 77.90.31.21:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173074734 (2h24m40.954008095s) [noise=true , address=77.90.31.10:4531 , piece_id=ZBSKCZZP6GCYJSKC5FUN5SYZSGC4JSHJBP6U2FAOBJOKIV2Z4UNQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: write tcp 159.223.143.39:46976->77.90.31.10:4531: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173092578 (2h24m40.954025958s) [noise=true , address=104.36.82.54:4166 , piece_id=P7ZZSYJMEKFEEPXPNPMB5PYOWMRVNNUH3KEFH5F7CNGZ4QMH3KOQ, seg_pos={0 0}, upload=2206]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 159.223.143.39:44726->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:25.173130344 (2h24m40.954063724s) [seg_pos={0 0}, upload=2206]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:16:25.198981863 (2h24m40.979915235s) [upload=2206]: Done uploading segments... Jul 27 00:16:25.199011535 (2h24m40.979944916s) [upload=2206]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.027808506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.] duration: 462.926429ms FAIL (2025-07-27 00:16:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.962737ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X] duration: 30.034506899s FAIL (2025-07-27 00:16:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034532237s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...XX..............X.X........X...XX........................] duration: 39.516002531s FAIL (2025-07-27 00:15:55 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.516028424s core uplink us-select-1 us1-placement-20

1
...
5503
5504
5505
5506
5507
...
6004