Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 362841 reports (Page 6764 of 7257)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.053780697s FAIL (2025-07-26 23:39:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.053796046s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..................XXX.X....XX.X.X.X.......XXX...XX.....X.XXX] duration: 30.000964872s FAIL (2025-07-26 23:39:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001009039s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.034295604s FAIL (2025-07-26 23:39:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== _SegmentBegin Jul 26 23:38:47.421931797 (1h47m3.202865179s) [seg_pos={0 0}, upload=1642]: Begin upload segment... Jul 26 23:38:47.421982422 (1h47m3.202915802s) [seg_pos={0 0}, upload=1642]: Done begin upload segment. Jul 26 23:38:47.422019079 (1h47m3.20295247s) [noise=true , address=104.36.82.7:4910 , piece_id=EGOKV45ZCHIFQ5IPJCYY6WLADSEPDTOCMQ3KQWN3ID3GWLY7SUAQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422027887 (1h47m3.202961269s) [noise=true , address=104.36.82.10:4077 , piece_id=IR4N6MKUEODMN2YFW3LVW75J7YCKYUTW5E2XQB5ZR7DZS2EJGGYA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422501337 (1h47m3.20343472s) [noise=true , address=45.91.116.13:4154 , piece_id=4PCIEOU663PW57PLYKKRXSQG4R3HMAQ4RUHA5ERJ57F6HP6L4UUA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422512233 (1h47m3.203445615s) [noise=true , address=104.36.82.16:4295 , piece_id=DBRMWHERYYRWIWRCCROBXXU5VU2AQP7VWWCB6EVTVRWPL5GAF4UA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422898572 (1h47m3.203831956s) [noise=true , address=104.36.82.10:4078 , piece_id=CD26AXN5DKNQIFS4OQDZ5L7O3OGJEWNAMGRXUU7QSWSKP7GPCH7A, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.423379839 (1h47m3.204313217s) [noise=true , address=104.36.82.11:4115 , piece_id=ZA5CQGTME7LRJBTJB6LTMFXMUUTFVBPVLAQX7MZJOCZSLD3APSDQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.423781818 (1h47m3.204715198s) [noise=true , address=45.91.116.9:4263 , piece_id=NFUKF5K2S26IU2A23DLFAWJYFR52RLRW64QDMYPE74UMIDGE4FUQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.424278599 (1h47m3.205211978s) [noise=true , address=104.36.82.7:4911 , piece_id=HEPDC2GPDJ4L5DLOUS6VJ3QQN3PQON3ZLDGPRIAC2YYDWRUFJALQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.424676105 (1h47m3.205609485s) [noise=true , address=45.91.116.6:4975 , piece_id=SXD3OD3KSUYOZ6QKDCX7C42CTPXB3WFUUFGJ4PZ4BUYXDMU5IA4Q, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425058266 (1h47m3.205991648s) [noise=true , address=45.91.116.4:4938 , piece_id=TR3BKO7T5XCOOIRVWQF5ZSCOJQDIAQUNM6E7L3HIZXMBKDYOXPCQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425439306 (1h47m3.206372674s) [noise=true , address=104.36.82.5:4838 , piece_id=D65E7MQKRI3U3PGSWANWGIED63D47XS4BNOBASXWYKF32I5AXYJQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425806572 (1h47m3.206739949s) [noise=true , address=104.36.82.15:4257 , piece_id=KMJXIMP4HBOKXCSTOUZE5HA7Z4CAYOPDDNKWXDMLKHVRVFJRV6MA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.426180477 (1h47m3.207113862s) [noise=true , address=77.90.31.11:4554 , piece_id=G4LS2EIJRIBYZIIX6TX7GYH2RKNKKFT2AOBRMG5XW3F22VP35OYA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.426580680 (1h47m3.207514049s) [noise=true , address=45.91.116.11:4119 , piece_id=PUWCXXG5QX7CARL7FGNC4QU7OHFQLPODH46ZJ5RRPDU4NHJRNYPA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.427600434 (1h47m3.208533815s) [noise=true , address=104.36.82.4:4802 , piece_id=OM3LPNKW4SVTZY5AFGGQAQ6DL3OU4AR7DLJZTSEMVX4PVQKBWW5A, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.428074992 (1h47m3.209008366s) [noise=true , address=104.36.82.9:4006 , piece_id=BSG5TOKXZJVZSMRPWXQDSQQBEQVM32BKDW6QZFGWM3SKQ7X3MRUQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.499282369 (1h47m3.280215738s) [upload=1642]: Next returned nil segment Jul 26 23:38:47.499299554 (1h47m3.280232934s) [upload=1642]: Waiting for error group managing segments... Jul 26 23:38:47.505148623 (1h47m3.286082s) [noise=true , address=45.91.116.6:4975 , piece_id=SXD3OD3KSUYOZ6QKDCX7C42CTPXB3WFUUFGJ4PZ4BUYXDMU5IA4Q, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.505300642 (1h47m3.286234012s) [noise=true , address=45.91.116.13:4154 , piece_id=4PCIEOU663PW57PLYKKRXSQG4R3HMAQ4RUHA5ERJ57F6HP6L4UUA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.505694113 (1h47m3.286627476s) [noise=true , address=45.91.116.4:4938 , piece_id=TR3BKO7T5XCOOIRVWQF5ZSCOJQDIAQUNM6E7L3HIZXMBKDYOXPCQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.658447490 (1h47m3.439380859s) [noise=true , address=45.91.116.11:4119 , piece_id=PUWCXXG5QX7CARL7FGNC4QU7OHFQLPODH46ZJ5RRPDU4NHJRNYPA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.953784843 (1h47m3.734718214s) [noise=true , address=45.91.116.9:4263 , piece_id=NFUKF5K2S26IU2A23DLFAWJYFR52RLRW64QDMYPE74UMIDGE4FUQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:39:17.347362137 (1h47m33.128295532s) [noise=true , address=104.36.82.16:4295 , piece_id=DBRMWHERYYRWIWRCCROBXXU5VU2AQP7VWWCB6EVTVRWPL5GAF4UA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347615763 (1h47m33.128549139s) [noise=true , address=104.36.82.10:4077 , piece_id=IR4N6MKUEODMN2YFW3LVW75J7YCKYUTW5E2XQB5ZR7DZS2EJGGYA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347692149 (1h47m33.128625575s) [noise=true , address=104.36.82.4:4802 , piece_id=OM3LPNKW4SVTZY5AFGGQAQ6DL3OU4AR7DLJZTSEMVX4PVQKBWW5A, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:60548->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347747760 (1h47m33.12868114s) [noise=true , address=77.90.31.11:4554 , piece_id=G4LS2EIJRIBYZIIX6TX7GYH2RKNKKFT2AOBRMG5XW3F22VP35OYA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 159.223.143.39:59656->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347812687 (1h47m33.128746075s) [noise=true , address=104.36.82.11:4115 , piece_id=ZA5CQGTME7LRJBTJB6LTMFXMUUTFVBPVLAQX7MZJOCZSLD3APSDQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347869793 (1h47m33.128803169s) [noise=true , address=104.36.82.7:4911 , piece_id=HEPDC2GPDJ4L5DLOUS6VJ3QQN3PQON3ZLDGPRIAC2YYDWRUFJALQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347928947 (1h47m33.128862311s) [noise=true , address=104.36.82.9:4006 , piece_id=BSG5TOKXZJVZSMRPWXQDSQQBEQVM32BKDW6QZFGWM3SKQ7X3MRUQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347973537 (1h47m33.128906913s) [noise=true , address=104.36.82.5:4838 , piece_id=D65E7MQKRI3U3PGSWANWGIED63D47XS4BNOBASXWYKF32I5AXYJQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347985423 (1h47m33.128918794s) [noise=true , address=104.36.82.10:4078 , piece_id=CD26AXN5DKNQIFS4OQDZ5L7O3OGJEWNAMGRXUU7QSWSKP7GPCH7A, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348010236 (1h47m33.128943603s) [noise=true , address=104.36.82.15:4257 , piece_id=KMJXIMP4HBOKXCSTOUZE5HA7Z4CAYOPDDNKWXDMLKHVRVFJRV6MA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348163130 (1h47m33.129096495s) [noise=true , address=104.36.82.7:4910 , piece_id=EGOKV45ZCHIFQ5IPJCYY6WLADSEPDTOCMQ3KQWN3ID3GWLY7SUAQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348240417 (1h47m33.129173795s) [seg_pos={0 0}, upload=1642]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:39:17.380145118 (1h47m33.161078487s) [upload=1642]: Done uploading segments... Jul 26 23:39:17.380169611 (1h47m33.161102993s) [upload=1642]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.034315477s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..........XX...X...X..XX.X...X..X.......X...X....XX.........] duration: 52.011418003s FAIL (2025-07-26 23:39:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.011453342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.048085333s FAIL (2025-07-26 23:39:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m_ObjectBegin Jul 26 23:38:10.284561719 (1h46m26.065495097s) [upload=1628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:38:10.321465178 (1h46m26.102398563s) [seg_pos={0 0}, upload=1628]: Begin upload segment... Jul 26 23:38:10.321530867 (1h46m26.102464232s) [seg_pos={0 0}, upload=1628]: Done begin upload segment. Jul 26 23:38:10.321704631 (1h46m26.102638011s) [noise=true , address=79.127.219.38:4005 , piece_id=QFE26MKRFQ436CZ7WGIXIP2BSKAOG4VOT74PSIXEJ3Z454L5TPIQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.321765633 (1h46m26.102699017s) [noise=true , address=79.127.205.246:4281 , piece_id=433ADJL66C72V6VEJD6MOHVRDCPVSUN6PNKG44QIXHHJ4T7FOETA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.322215816 (1h46m26.103149198s) [noise=true , address=104.36.82.20:4453 , piece_id=BU3NIFK2J3Q3RTUZSKGGTRBOOKQ3KTJLTJR55IEHYB7PVKADMKXQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.323171821 (1h46m26.104105208s) [noise=true , address=77.90.31.19:4849 , piece_id=PATHA4GHVGGHJAQM4UTUIXAUZLPW3ZT3ZDIVK6IAWPIX6WP2VBUA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.323866994 (1h46m26.10480038s) [noise=true , address=79.127.201.215:4350 , piece_id=NYGBEM55XMQDXVQWM33QXCOB536IQLZKJA2I27ANYBUWZJUQ7YXA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.324688778 (1h46m26.105622162s) [noise=true , address=104.36.82.55:4212 , piece_id=ZOSGWHZY2QTNZZV6X4ADRHD57JTXRQJEKDTOIQXVH5K5TGYLSTYA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.325242287 (1h46m26.106175665s) [noise=true , address=79.127.205.233:4643 , piece_id=UTCZTZ4O2LCRKJUMJYOHY57WWPL55ARG36WCGKEVGCT5JR5EAN6Q, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.326044037 (1h46m26.106977422s) [noise=true , address=79.127.219.47:4069 , piece_id=OWQIVWCCMKXX76ZRBQRRXBTJLBAEQEAKPPEYVBLN7QMBDKVVEONA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.326676184 (1h46m26.107609567s) [noise=true , address=104.36.82.46:4513 , piece_id=GIVUVGS6ZLJ4WWMQZM6HVQPHGMTPWYGOZPUZGFCNRYT43VLWESXA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.327510134 (1h46m26.108443522s) [noise=true , address=77.90.31.16:4760 , piece_id=DO5T3BLSIMPKZCTSM23PXWIVIDMCZUCXHLFYSNLXKYRUSRGUEOWQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.327944372 (1h46m26.108877754s) [noise=true , address=79.127.220.98:4422 , piece_id=7HA5PLJLPSTA5BNYAV44IILMURZW7C3Y7F5ZT3NOU5KP52PGO5YA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.328431350 (1h46m26.109364735s) [noise=true , address=79.127.205.234:4616 , piece_id=FYGIXCXURVOZ6EJHBH7DGYQAAS3EGB6DK46SQMWGTJ4X2CZLTH2Q, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.328707695 (1h46m26.109641079s) [noise=true , address=64.49.13.140:4508 , piece_id=XFBJZWZM55CRPX3NGSF25NKTV7ESPPQFLRPWE537IH2R6BO5H7QQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329331982 (1h46m26.110265367s) [noise=true , address=64.49.13.138:4482 , piece_id=RWLMUBKZZUWF5YXKB3O427MQA7GLM3YX2V56F7QOSYMM4GD2J6KA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329445281 (1h46m26.110378667s) [noise=true , address=104.36.82.21:4474 , piece_id=YITVHQJZ4IJ6EU2QRMUUEEHEG5WI6PLMBPA5M4NUD4PVB4I6VWFQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329904045 (1h46m26.110837431s) [noise=true , address=104.36.82.53:4143 , piece_id=NBVFBFTQNWR73UR4JYLVJ36KLVCYK3CI6QDZLIB6G63SXYXGUWYA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.330629252 (1h46m26.111562635s) [noise=true , address=79.127.205.235:4690 , piece_id=BOL6OJND5T3EKPL6V4AWKX3H6V276L332OXMV7MWFWMWDMU6T2UQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.331398742 (1h46m26.112332129s) [noise=true , address=104.36.82.47:4541 , piece_id=VM6H3II5HCO6A56L5LXCDPZN2T6B5DJWVZEURQJGTTFHVBOLFMGQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.415942595 (1h46m26.196875972s) [upload=1628]: Next returned nil segment Jul 26 23:38:10.415969697 (1h46m26.196903072s) [upload=1628]: Waiting for error group managing segments... Jul 26 23:38:10.423421258 (1h46m26.20435464s) [noise=true , address=79.127.219.38:4005 , piece_id=QFE26MKRFQ436CZ7WGIXIP2BSKAOG4VOT74PSIXEJ3Z454L5TPIQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.450413342 (1h46m26.231346723s) [noise=true , address=77.90.31.16:4760 , piece_id=DO5T3BLSIMPKZCTSM23PXWIVIDMCZUCXHLFYSNLXKYRUSRGUEOWQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.507513144 (1h46m26.288446509s) [noise=true , address=79.127.220.98:4422 , piece_id=7HA5PLJLPSTA5BNYAV44IILMURZW7C3Y7F5ZT3NOU5KP52PGO5YA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.635830264 (1h46m26.416763631s) [noise=true , address=79.127.219.47:4069 , piece_id=OWQIVWCCMKXX76ZRBQRRXBTJLBAEQEAKPPEYVBLN7QMBDKVVEONA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.885921183 (1h46m26.666854544s) [noise=true , address=79.127.205.235:4690 , piece_id=BOL6OJND5T3EKPL6V4AWKX3H6V276L332OXMV7MWFWMWDMU6T2UQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.905141234 (1h46m26.686074603s) [noise=true , address=79.127.205.246:4281 , piece_id=433ADJL66C72V6VEJD6MOHVRDCPVSUN6PNKG44QIXHHJ4T7FOETA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.945685357 (1h46m26.726618725s) [noise=true , address=79.127.205.233:4643 , piece_id=UTCZTZ4O2LCRKJUMJYOHY57WWPL55ARG36WCGKEVGCT5JR5EAN6Q, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:11.345380090 (1h46m27.126313466s) [noise=true , address=79.127.205.234:4616 , piece_id=FYGIXCXURVOZ6EJHBH7DGYQAAS3EGB6DK46SQMWGTJ4X2CZLTH2Q, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:14.304790376 (1h46m30.085723752s) [noise=true , address=64.49.13.140:4508 , piece_id=XFBJZWZM55CRPX3NGSF25NKTV7ESPPQFLRPWE537IH2R6BO5H7QQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:38:14.944937304 (1h46m30.725870692s) [noise=true , address=64.49.13.138:4482 , piece_id=RWLMUBKZZUWF5YXKB3O427MQA7GLM3YX2V56F7QOSYMM4GD2J6KA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:20.402234579 (1h46m36.183167948s) [noise=true , address=77.90.31.19:4849 , piece_id=PATHA4GHVGGHJAQM4UTUIXAUZLPW3ZT3ZDIVK6IAWPIX6WP2VBUA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:26.046187863 (1h46m41.827121239s) [noise=true , address=79.127.201.215:4350 , piece_id=NYGBEM55XMQDXVQWM33QXCOB536IQLZKJA2I27ANYBUWZJUQ7YXA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:39:10.252609249 (1h47m26.033542629s) [noise=true , address=104.36.82.53:4143 , piece_id=NBVFBFTQNWR73UR4JYLVJ36KLVCYK3CI6QDZLIB6G63SXYXGUWYA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252723229 (1h47m26.033656605s) [noise=true , address=104.36.82.47:4541 , piece_id=VM6H3II5HCO6A56L5LXCDPZN2T6B5DJWVZEURQJGTTFHVBOLFMGQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252804093 (1h47m26.033737469s) [noise=true , address=104.36.82.55:4212 , piece_id=ZOSGWHZY2QTNZZV6X4ADRHD57JTXRQJEKDTOIQXVH5K5TGYLSTYA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: rpc: dial tcp 104.36.82.55:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252882501 (1h47m26.03381588s) [noise=true , address=104.36.82.21:4474 , piece_id=YITVHQJZ4IJ6EU2QRMUUEEHEG5WI6PLMBPA5M4NUD4PVB4I6VWFQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:104.36.82.21:4474): protocol: rpc: dial tcp 104.36.82.21:4474: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252940420 (1h47m26.033873797s) [noise=true , address=104.36.82.20:4453 , piece_id=BU3NIFK2J3Q3RTUZSKGGTRBOOKQ3KTJLTJR55IEHYB7PVKADMKXQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1qxsiscygr1N83dQS4UE53NJSRcLKsAtuDRCuk7ZXZQ99Yswhi, address:104.36.82.20:4453): protocol: rpc: dial tcp 104.36.82.20:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.253019370 (1h47m26.033952748s) [noise=true , address=104.36.82.46:4513 , piece_id=GIVUVGS6ZLJ4WWMQZM6HVQPHGMTPWYGOZPUZGFCNRYT43VLWESXA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:59284->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.253058304 (1h47m26.033991686s) [seg_pos={0 0}, upload=1628]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:39:10.297369201 (1h47m26.07830257s) [upload=1628]: Done uploading segments... Jul 26 23:39:10.297408551 (1h47m26.078341925s) [upload=1628]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.048114006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [.............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026301846s FAIL (2025-07-26 23:39:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.026330682s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032727023s FAIL (2025-07-26 23:38:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 26 23:38:17.282381672 (1h46m33.063315054s) [upload=1630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:38:17.282388081 (1h46m33.063321456s) [upload=1630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:38:17.325457222 (1h46m33.106390598s) [seg_pos={0 0}, upload=1630]: Begin upload segment... Jul 26 23:38:17.325499025 (1h46m33.10643239s) [seg_pos={0 0}, upload=1630]: Done begin upload segment. Jul 26 23:38:17.325522894 (1h46m33.106456275s) [noise=true , address=77.90.31.16:4734 , piece_id=GHJRLYE6AXZAEQKZ4A3R73GQ3B6DXRGX3VGFD7TQ2LOZRL3V7GVA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326015952 (1h46m33.106949334s) [noise=true , address=104.36.82.8:4971 , piece_id=7RLCRKTGMDQZ6E5NTU442EUZ2WTXINGMEWJ7Y35QUFHXPAS53C3A, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326503029 (1h46m33.10743641s) [noise=true , address=45.91.116.6:4975 , piece_id=NJ7MS77HHV3HZS5YLWK35DYMKXRHNYEBOIIITNODINWATDDABFZQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326863575 (1h46m33.10779695s) [noise=true , address=45.91.116.10:4082 , piece_id=MINL34U236G4ZORSPX45HRZFS3PRH4NWNYILCJFE2WOTSUCOJUGA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.327943710 (1h46m33.108877095s) [noise=true , address=104.36.82.7:4912 , piece_id=PPLQXHEUC7RBYND6GPPBMUOOAKDFQZ4K3HKVMNBEEWNEKI2VPUDA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.328763942 (1h46m33.109697321s) [noise=true , address=45.91.116.12:4225 , piece_id=6DAA3NVIJWRYLPUT7GIOWRRFHPTSNJWOHLSKQPMFSWDZB6VBKHMA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329450902 (1h46m33.110384285s) [noise=true , address=104.36.82.9:4005 , piece_id=J3X3ZUFI7WYT5AOE7R6F7FWS7RWWNDFHRVWJXHYRO2IXJJT4RBNA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329827291 (1h46m33.110760669s) [noise=true , address=104.36.82.12:4150 , piece_id=SLRKLGRC3N5DQ56ICKET243DWDXYOS6KKZPQLQDHXOZEIJ3UONUA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329914440 (1h46m33.110847824s) [noise=true , address=45.91.116.9:4263 , piece_id=NR7ROCNVY76S5TRN2IP3QUVZAP7RORIII4ATL2QWU7VN4EKDG3CQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.330382271 (1h46m33.111315649s) [noise=true , address=104.36.82.4:4804 , piece_id=CXGM45OCDQ23LJ3UXOMIR46V5QONEDF7E4DSGKHJLP3SEZ23GMDQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.330842660 (1h46m33.111776042s) [noise=true , address=45.91.116.4:4937 , piece_id=TARXCM4JN5PYNBULYPBZWE53Y4WPV2ZIWIN4SCSBEI6VCO55S5FA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.331249859 (1h46m33.112183237s) [noise=true , address=104.36.82.2:4729 , piece_id=LRX7X6MZISWVBGG6FSLGBSAJXQDWYF6QXNBPR3SW3AQCNNO4DL2Q, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.331700245 (1h46m33.112633631s) [noise=true , address=104.36.82.10:4078 , piece_id=UNO7ZD6BN67INUQFRZEMLVIQVFLJMZBBJEQJISDZYWK6RUQVVLGA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.332193010 (1h46m33.113126396s) [noise=true , address=104.36.82.12:4151 , piece_id=4JSTZ7WBRM3ZMO3MHMVJDIZZPKKNGN4LBVG24GGOI6IA7NFU7MMA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.332877279 (1h46m33.113810664s) [noise=true , address=104.36.82.6:4875 , piece_id=OGMKDBNZUM5C456KDEMQBL233ZG4STKKG2UPWY6YELTFXSCRCEUA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.333406079 (1h46m33.114339466s) [noise=true , address=104.36.82.11:4115 , piece_id=24X2WT64KHLE3FJVGI5CKXQMUWDVEDAWA4PHU3EFNRLH7XKQ5CUQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.415164740 (1h46m33.19609812s) [upload=1630]: Next returned nil segment Jul 26 23:38:17.415188443 (1h46m33.19612183s) [upload=1630]: Waiting for error group managing segments... Jul 26 23:38:17.419552774 (1h46m33.200486142s) [noise=true , address=45.91.116.4:4937 , piece_id=TARXCM4JN5PYNBULYPBZWE53Y4WPV2ZIWIN4SCSBEI6VCO55S5FA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.420179959 (1h46m33.201113325s) [noise=true , address=45.91.116.6:4975 , piece_id=NJ7MS77HHV3HZS5YLWK35DYMKXRHNYEBOIIITNODINWATDDABFZQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.420724133 (1h46m33.201657498s) [noise=true , address=45.91.116.10:4082 , piece_id=MINL34U236G4ZORSPX45HRZFS3PRH4NWNYILCJFE2WOTSUCOJUGA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.421161211 (1h46m33.202094586s) [noise=true , address=45.91.116.12:4225 , piece_id=6DAA3NVIJWRYLPUT7GIOWRRFHPTSNJWOHLSKQPMFSWDZB6VBKHMA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.421636875 (1h46m33.202570244s) [noise=true , address=77.90.31.16:4734 , piece_id=GHJRLYE6AXZAEQKZ4A3R73GQ3B6DXRGX3VGFD7TQ2LOZRL3V7GVA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.483748356 (1h46m33.264681732s) [noise=true , address=45.91.116.9:4263 , piece_id=NR7ROCNVY76S5TRN2IP3QUVZAP7RORIII4ATL2QWU7VN4EKDG3CQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:47.247103928 (1h47m3.028037302s) [noise=true , address=104.36.82.12:4150 , piece_id=SLRKLGRC3N5DQ56ICKET243DWDXYOS6KKZPQLQDHXOZEIJ3UONUA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247174663 (1h47m3.028108028s) [noise=true , address=104.36.82.4:4804 , piece_id=CXGM45OCDQ23LJ3UXOMIR46V5QONEDF7E4DSGKHJLP3SEZ23GMDQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247208890 (1h47m3.02814228s) [noise=true , address=104.36.82.10:4078 , piece_id=UNO7ZD6BN67INUQFRZEMLVIQVFLJMZBBJEQJISDZYWK6RUQVVLGA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247224212 (1h47m3.028157578s) [noise=true , address=104.36.82.2:4729 , piece_id=LRX7X6MZISWVBGG6FSLGBSAJXQDWYF6QXNBPR3SW3AQCNNO4DL2Q, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247274888 (1h47m3.028208253s) [noise=true , address=104.36.82.8:4971 , piece_id=7RLCRKTGMDQZ6E5NTU442EUZ2WTXINGMEWJ7Y35QUFHXPAS53C3A, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247297158 (1h47m3.028230534s) [noise=true , address=104.36.82.9:4005 , piece_id=J3X3ZUFI7WYT5AOE7R6F7FWS7RWWNDFHRVWJXHYRO2IXJJT4RBNA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247325516 (1h47m3.028258881s) [noise=true , address=104.36.82.7:4912 , piece_id=PPLQXHEUC7RBYND6GPPBMUOOAKDFQZ4K3HKVMNBEEWNEKI2VPUDA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:42306->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247363193 (1h47m3.028296558s) [noise=true , address=104.36.82.6:4875 , piece_id=OGMKDBNZUM5C456KDEMQBL233ZG4STKKG2UPWY6YELTFXSCRCEUA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247388224 (1h47m3.028321593s) [noise=true , address=104.36.82.12:4151 , piece_id=4JSTZ7WBRM3ZMO3MHMVJDIZZPKKNGN4LBVG24GGOI6IA7NFU7MMA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247425257 (1h47m3.02835863s) [noise=true , address=104.36.82.11:4115 , piece_id=24X2WT64KHLE3FJVGI5CKXQMUWDVEDAWA4PHU3EFNRLH7XKQ5CUQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:40042->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247465754 (1h47m3.028399139s) [seg_pos={0 0}, upload=1630]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:38:47.277303759 (1h47m3.05823714s) [upload=1630]: Done uploading segments... Jul 26 23:38:47.277332606 (1h47m3.058265987s) [upload=1630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.03275199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [..............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.035724491s FAIL (2025-07-26 23:38:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.035751986s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...................XXX.X....XX.X.X.X.......XXX...XX.....X.XX] duration: 30.000537323s FAIL (2025-07-26 23:38:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000551226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.209847972s FAIL (2025-07-26 23:38:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:37:05.096773169 (1h45m20.877706558s) [seg_pos={0 0}, upload=1612]: Begin upload segment... Jul 26 23:37:05.096857219 (1h45m20.877790586s) [seg_pos={0 0}, upload=1612]: Done begin upload segment. Jul 26 23:37:05.096908307 (1h45m20.877841696s) [noise=true , address=79.127.220.101:4048 , piece_id=BNVOMJPTRGIBZCZLGMTZAZ65E4V75ZYH57VUMRWPSBXTOA4J3ALQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.097588314 (1h45m20.878521704s) [noise=true , address=79.127.201.215:4344 , piece_id=OO7W7TLFHLYV5KYDOU4BTKW4IFFKZ7O3GI7OPFBL6ESMZ4WJHYDQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.098136937 (1h45m20.879070327s) [noise=true , address=79.127.205.245:4232 , piece_id=SOPDQGL2JIAUTDZLMLC3PR4U5B3QZA73537GYCO47K4MZMBOMPAQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.098661917 (1h45m20.879595294s) [noise=true , address=79.127.219.38:4010 , piece_id=ZW4RFLLDM5P37SC4AK3ZXRIZW3GDXUZB4FLKYHCKW43RHM37WOOQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.099218376 (1h45m20.880151765s) [noise=true , address=104.36.82.46:4489 , piece_id=6REVHFFE7A5Z6U3T5UGIIZQU6NKA6E6GMBGUZTGFDNOBO557MTTQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.099720240 (1h45m20.880653634s) [noise=true , address=104.36.82.14:4255 , piece_id=25QTE3D4J2FIS73QEMHHBTJA5RGJGUUIGTKP47C4M6ZXU77ZZ5RQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.114224946 (1h45m20.895158571s) [noise=true , address=104.36.82.47:4537 , piece_id=GDEAWPJQQAGMDCPNNAS22OCVX4JUMLRI5JSL6GURTRJLACOPZN6A, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.114755110 (1h45m20.895688499s) [noise=true , address=79.127.205.234:4613 , piece_id=EV4IC5KS4G7ZKTRXUPNCDUW4PPJB4U4H75KPQLNJ73CXPNLGUYBQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.115253358 (1h45m20.896186739s) [noise=true , address=45.91.116.14:4206 , piece_id=TOFCQ2KD7AB4ANQLNXE4KTME2CWGQXXYZKOM5CVZYTZC5NA6IJKA, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.115620862 (1h45m20.896554231s) [noise=true , address=45.91.116.5:4306 , piece_id=UKT6IJ57R6SNKAI5ZBJC3WRRTULAFLTW3AG6WH5BVOA25XRHQP4Q, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.116183317 (1h45m20.897116706s) [noise=true , address=79.127.205.246:4288 , piece_id=KN2CT2T6SYXEB5IVJKUFVCPB7ESHUPHSZE5NL457YIRIRG3S5XFQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.116667914 (1h45m20.897601299s) [noise=true , address=104.36.82.56:4253 , piece_id=WGOYQBXJPUMYQXQW5ATXNZ6K4FS55EIFCYWDB3YNQ2B6UHCHJXDA, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.117658564 (1h45m20.89859195s) [noise=true , address=156.146.43.225:4434 , piece_id=PRVUI6LFEJJVEQNU4PTEZ6LIVKOE6E6XBBDSZHJJOYNKOPIKR4JA, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.118129485 (1h45m20.899062874s) [noise=true , address=45.91.116.12:4230 , piece_id=IHLW7BEYQOH2J3WZOAA33B7D7D2CYS5VKWFPZ4A3OFOQIY5EQV3A, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.118583519 (1h45m20.899516887s) [noise=true , address=79.127.205.233:4639 , piece_id=V6QFNCYNVTYXYY72RT2RSFCSYAC65JNNDNQTUSKGDGSRF4SGSWEQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.119004357 (1h45m20.899937746s) [noise=true , address=104.36.82.55:4195 , piece_id=HEUPCCF7MQ2CUFN2BAUXSUQLDQTFN3NF6VI4GQOYL6FXYNWF4MMQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.119365902 (1h45m20.900299289s) [noise=true , address=45.91.116.14:4194 , piece_id=77YICZE7GUHW6RHRTIHRPM36BMYWSVRX5R34VBYBFNOLD5RVS6LQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.119798901 (1h45m20.900732285s) [noise=true , address=104.36.82.52:4115 , piece_id=HQFSWNP2CPBSATF6XU3E3UINJWV627VBA75JT33V7LAMDGKJWXNQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.173071095 (1h45m20.95400448s) [upload=1612]: Next returned nil segment Jul 26 23:37:05.173093064 (1h45m20.954026449s) [upload=1612]: Waiting for error group managing segments... Jul 26 23:37:05.177523410 (1h45m20.958456788s) [noise=true , address=79.127.219.38:4010 , piece_id=ZW4RFLLDM5P37SC4AK3ZXRIZW3GDXUZB4FLKYHCKW43RHM37WOOQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.215155675 (1h45m20.996089056s) [noise=true , address=79.127.220.101:4048 , piece_id=BNVOMJPTRGIBZCZLGMTZAZ65E4V75ZYH57VUMRWPSBXTOA4J3ALQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:37:05.628395718 (1h45m21.40932909s) [noise=true , address=45.91.116.14:4194 , piece_id=77YICZE7GUHW6RHRTIHRPM36BMYWSVRX5R34VBYBFNOLD5RVS6LQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.712600929 (1h45m21.493534298s) [noise=true , address=79.127.205.233:4639 , piece_id=V6QFNCYNVTYXYY72RT2RSFCSYAC65JNNDNQTUSKGDGSRF4SGSWEQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.733477744 (1h45m21.514411113s) [noise=true , address=79.127.205.245:4232 , piece_id=SOPDQGL2JIAUTDZLMLC3PR4U5B3QZA73537GYCO47K4MZMBOMPAQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.787828945 (1h45m21.568762315s) [noise=true , address=79.127.205.246:4288 , piece_id=KN2CT2T6SYXEB5IVJKUFVCPB7ESHUPHSZE5NL457YIRIRG3S5XFQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.844604883 (1h45m21.62553826s) [noise=true , address=79.127.205.234:4613 , piece_id=EV4IC5KS4G7ZKTRXUPNCDUW4PPJB4U4H75KPQLNJ73CXPNLGUYBQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.929656818 (1h45m21.710590181s) [noise=true , address=45.91.116.5:4306 , piece_id=UKT6IJ57R6SNKAI5ZBJC3WRRTULAFLTW3AG6WH5BVOA25XRHQP4Q, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:06.351490290 (1h45m22.132423658s) [noise=true , address=79.127.201.215:4344 , piece_id=OO7W7TLFHLYV5KYDOU4BTKW4IFFKZ7O3GI7OPFBL6ESMZ4WJHYDQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:08.511765295 (1h45m24.292698674s) [noise=true , address=156.146.43.225:4434 , piece_id=PRVUI6LFEJJVEQNU4PTEZ6LIVKOE6E6XBBDSZHJJOYNKOPIKR4JA, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:09.818107843 (1h45m25.599041227s) [noise=true , address=45.91.116.14:4206 , piece_id=TOFCQ2KD7AB4ANQLNXE4KTME2CWGQXXYZKOM5CVZYTZC5NA6IJKA, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:12.545384021 (1h45m28.326317398s) [noise=true , address=45.91.116.12:4230 , piece_id=IHLW7BEYQOH2J3WZOAA33B7D7D2CYS5VKWFPZ4A3OFOQIY5EQV3A, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:38:04.836492282 (1h46m20.617425667s) [noise=true , address=104.36.82.46:4489 , piece_id=6REVHFFE7A5Z6U3T5UGIIZQU6NKA6E6GMBGUZTGFDNOBO557MTTQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836547165 (1h46m20.617480559s) [noise=true , address=104.36.82.55:4195 , piece_id=HEUPCCF7MQ2CUFN2BAUXSUQLDQTFN3NF6VI4GQOYL6FXYNWF4MMQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: rpc: dial tcp 104.36.82.55:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836612842 (1h46m20.617546221s) [noise=true , address=104.36.82.14:4255 , piece_id=25QTE3D4J2FIS73QEMHHBTJA5RGJGUUIGTKP47C4M6ZXU77ZZ5RQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: write tcp 159.223.143.39:57756->104.36.82.14:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836628262 (1h46m20.617561627s) [noise=true , address=104.36.82.47:4537 , piece_id=GDEAWPJQQAGMDCPNNAS22OCVX4JUMLRI5JSL6GURTRJLACOPZN6A, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836685559 (1h46m20.61761905s) [noise=true , address=104.36.82.56:4253 , piece_id=WGOYQBXJPUMYQXQW5ATXNZ6K4FS55EIFCYWDB3YNQ2B6UHCHJXDA, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 159.223.143.39:35020->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836698859 (1h46m20.617632238s) [noise=true , address=104.36.82.52:4115 , piece_id=HQFSWNP2CPBSATF6XU3E3UINJWV627VBA75JT33V7LAMDGKJWXNQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: write tcp 159.223.143.39:57942->104.36.82.52:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836722705 (1h46m20.617656088s) [seg_pos={0 0}, upload=1612]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:38:05.043294997 (1h46m20.824228366s) [upload=1612]: Done uploading segments... Jul 26 23:38:05.043324151 (1h46m20.824257519s) [upload=1612]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.209872524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....................XXX.X....XX.X.X.X.......XXX...XX.....X.X] duration: 30.00074243s FAIL (2025-07-26 23:37:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000839161s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [...............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.033373309s FAIL (2025-07-26 23:37:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.033414766s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X..........XX...X...X..XX.X...X..X.......X...X....XX........] duration: 7.687785762s FAIL (2025-07-26 23:37:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.687814176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026265514s FAIL (2025-07-26 23:37:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== XKRW2LU24SA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:56724->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914569057 (1h44m39.69550244s) [seg_pos={0 0}, upload=1595]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:36:23.942316640 (1h44m39.723250007s) [upload=1595]: Done uploading segments... Jul 26 23:36:23.942360746 (1h44m39.723294114s) [upload=1595]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 23:37:05.883283406 (1h45m21.664216789s) [upload=1614]: Starting upload Jul 26 23:37:05.883396987 (1h45m21.664330374s) [upload=1614]: Done starting upload Jul 26 23:37:05.883550745 (1h45m21.664484121s) [upload=1614]: Uploading segments... Jul 26 23:37:05.885260248 (1h45m21.666193626s) [upload=1614]: Got next segment. Inline: false Jul 26 23:37:05.885374047 (1h45m21.666307429s) [upload=1614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:37:05.885381923 (1h45m21.666315306s) [upload=1614]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:37:05.916570243 (1h45m21.697503656s) [seg_pos={0 0}, upload=1614]: Begin upload segment... Jul 26 23:37:05.916642029 (1h45m21.697575415s) [seg_pos={0 0}, upload=1614]: Done begin upload segment. Jul 26 23:37:05.916791900 (1h45m21.697725286s) [noise=true , address=104.36.82.5:4837 , piece_id=3ZGRR77BNNACN36CM4LJMEVA7DWAKPBCYZAR4XLOCLKHHXUYT5DA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.916848880 (1h45m21.697782271s) [noise=true , address=45.91.116.4:4938 , piece_id=PKRA7ZRCFLYXG6AKWBNQ65SUP5WCIAC7RM44VVDKMZNLC6T65SXQ, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.917352874 (1h45m21.69828626s) [noise=true , address=77.90.31.16:4733 , piece_id=EWDITDG3E4XDBVIFNKEMLGELY3NSKUFMEIE3RLFN7T26JNWNDSFA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.918915294 (1h45m21.699848675s) [noise=true , address=45.91.116.13:4154 , piece_id=GZKEJNBYNPOMK76SW4KMLS6EFIFL3QV2VSEX2LPPFOAQKTDTZS6Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.919768361 (1h45m21.70070175s) [noise=true , address=45.91.116.8:4048 , piece_id=7YZCLJRYL7PKTVEQNGYJK3JEGDUSMYWMPAVQ2GZBKPTZIHPM7ZTA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.920640421 (1h45m21.701573807s) [noise=true , address=77.90.31.13:4628 , piece_id=CGP4ZJZVXYMAQXRVO2SX3EVRLK7YDSSE64TCUPYTDKA7M3LUCOTA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.923212326 (1h45m21.704145715s) [noise=true , address=104.36.82.12:4149 , piece_id=LXJWCOI45R6XRQSG63SXPDYIJGWR3FZXVRCRVDGYZ25CHLBJ3FPA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.924383087 (1h45m21.70531647s) [noise=true , address=77.90.31.18:4806 , piece_id=SIIJCEA2EU7KRYV3Y4NSHKCDQBY2P7GTJKDVM3ARWJKK5Q5U5TCA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.925189703 (1h45m21.706123094s) [noise=true , address=77.90.31.19:4843 , piece_id=B5NR7IJUVLK4OIROD63FVMUH764RIM4C4UL5H5RAEMKUNRKQK62Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.926252181 (1h45m21.70718557s) [noise=true , address=104.36.82.7:4912 , piece_id=D5VCAUEGBTZCY75OA225RQGNH2B2Q23JFI5363G6SB7F4HUX347Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.927080241 (1h45m21.708013631s) [noise=true , address=77.90.31.10:4518 , piece_id=Y36RWUQBORNLUQKMEDREYKK6ED6PV4TTRXZCHBGV2PW2MESWPBIA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.928100694 (1h45m21.709034124s) [noise=true , address=45.91.116.7:4009 , piece_id=YAPWDQTQA4AL6VAFUPTKAJUCM7SBXF3JJND6HMBZBZ5BCIDFJ5VQ, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.929246960 (1h45m21.710180343s) [noise=true , address=45.91.116.5:4299 , piece_id=3HHZWA4NMPPFXAIANZ6CE26Z2XXL3AHYHLQZ7L4PC526WXAS6S4Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.929705326 (1h45m21.710638712s) [noise=true , address=104.36.82.6:4873 , piece_id=MEQ3OR2ZVQ6RUH35J5QXW3SME7VFQVZH7EGKPKOPB77CUFB2YIIA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.930074181 (1h45m21.711007569s) [noise=true , address=104.36.82.4:4804 , piece_id=53JXL5ECNNK2CYGWVV55UOBJRZNU6NNFXYFSPH2VC4E4DO3QA24A, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.930456851 (1h45m21.711390239s) [noise=true , address=45.91.116.12:4227 , piece_id=WJKUAZQKDDGTYYPXWGNBWGCKJUM7OD3ZMPAP6BILZR7OI5ZHQ44A, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:06.054626560 (1h45m21.835559922s) [upload=1614]: Next returned nil segment Jul 26 23:37:06.054646894 (1h45m21.835580258s) [upload=1614]: Waiting for error group managing segments... Jul 26 23:37:06.060866261 (1h45m21.841799625s) [noise=true , address=45.91.116.4:4938 , piece_id=PKRA7ZRCFLYXG6AKWBNQ65SUP5WCIAC7RM44VVDKMZNLC6T65SXQ, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.061930323 (1h45m21.842863686s) [noise=true , address=45.91.116.7:4009 , piece_id=YAPWDQTQA4AL6VAFUPTKAJUCM7SBXF3JJND6HMBZBZ5BCIDFJ5VQ, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.064180345 (1h45m21.845113714s) [noise=true , address=77.90.31.18:4806 , piece_id=SIIJCEA2EU7KRYV3Y4NSHKCDQBY2P7GTJKDVM3ARWJKK5Q5U5TCA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.065098049 (1h45m21.846031419s) [noise=true , address=45.91.116.8:4048 , piece_id=7YZCLJRYL7PKTVEQNGYJK3JEGDUSMYWMPAVQ2GZBKPTZIHPM7ZTA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.066996167 (1h45m21.847929538s) [noise=true , address=45.91.116.13:4154 , piece_id=GZKEJNBYNPOMK76SW4KMLS6EFIFL3QV2VSEX2LPPFOAQKTDTZS6Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.071454152 (1h45m21.852387528s) [noise=true , address=45.91.116.12:4227 , piece_id=WJKUAZQKDDGTYYPXWGNBWGCKJUM7OD3ZMPAP6BILZR7OI5ZHQ44A, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.112162673 (1h45m21.893096052s) [noise=true , address=77.90.31.16:4733 , piece_id=EWDITDG3E4XDBVIFNKEMLGELY3NSKUFMEIE3RLFN7T26JNWNDSFA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.138091162 (1h45m21.919024535s) [noise=true , address=45.91.116.5:4299 , piece_id=3HHZWA4NMPPFXAIANZ6CE26Z2XXL3AHYHLQZ7L4PC526WXAS6S4Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:07.464697484 (1h45m23.245630855s) [noise=true , address=77.90.31.19:4843 , piece_id=B5NR7IJUVLK4OIROD63FVMUH764RIM4C4UL5H5RAEMKUNRKQK62Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:07.490168140 (1h45m23.271101516s) [noise=true , address=77.90.31.13:4628 , piece_id=CGP4ZJZVXYMAQXRVO2SX3EVRLK7YDSSE64TCUPYTDKA7M3LUCOTA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:07.518510416 (1h45m23.299443795s) [noise=true , address=77.90.31.10:4518 , piece_id=Y36RWUQBORNLUQKMEDREYKK6ED6PV4TTRXZCHBGV2PW2MESWPBIA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:35.853485599 (1h45m51.634418966s) [noise=true , address=104.36.82.4:4804 , piece_id=53JXL5ECNNK2CYGWVV55UOBJRZNU6NNFXYFSPH2VC4E4DO3QA24A, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:47272->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853562766 (1h45m51.634496142s) [noise=true , address=104.36.82.5:4837 , piece_id=3ZGRR77BNNACN36CM4LJMEVA7DWAKPBCYZAR4XLOCLKHHXUYT5DA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853618022 (1h45m51.634551388s) [noise=true , address=104.36.82.6:4873 , piece_id=MEQ3OR2ZVQ6RUH35J5QXW3SME7VFQVZH7EGKPKOPB77CUFB2YIIA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:47036->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853669494 (1h45m51.63460286s) [noise=true , address=104.36.82.12:4149 , piece_id=LXJWCOI45R6XRQSG63SXPDYIJGWR3FZXVRCRVDGYZ25CHLBJ3FPA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:51778->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853721476 (1h45m51.63465485s) [noise=true , address=104.36.82.7:4912 , piece_id=D5VCAUEGBTZCY75OA225RQGNH2B2Q23JFI5363G6SB7F4HUX347Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853743014 (1h45m51.634676391s) [seg_pos={0 0}, upload=1614]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:37:35.877769449 (1h45m51.658702812s) [upload=1614]: Done uploading segments... Jul 26 23:37:35.877802051 (1h45m51.658735432s) [upload=1614]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.026293224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035030859s FAIL (2025-07-26 23:37:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1h44m20.573930382s) [upload=1598]: Uploading segments... Jul 26 23:36:04.793326392 (1h44m20.574259777s) [upload=1598]: Got next segment. Inline: false Jul 26 23:36:04.793422468 (1h44m20.574355851s) [upload=1598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:36:04.793431673 (1h44m20.574365059s) [upload=1598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:36:04.839132394 (1h44m20.62006578s) [seg_pos={0 0}, upload=1598]: Begin upload segment... Jul 26 23:36:04.839230235 (1h44m20.620163625s) [noise=true , address=104.36.82.54:4187 , piece_id=BIWDUNNMQZGGWA7P3HOHLVEWC3LY7UUWORMY6LIOK7UN55BA55BQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.839744212 (1h44m20.620677596s) [noise=true , address=79.127.219.51:4196 , piece_id=U4NDAUEAXGWNBALDD32JBGNQLZQPE4T6IILT7JPWUE6WWNMYFBCQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.840339521 (1h44m20.621272911s) [noise=true , address=79.127.205.245:4237 , piece_id=WGJX7TMGFWAS3ECOHZXNTNUKDXGDCYQX4RFJU2LTHHC3OE335DGQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.840697723 (1h44m20.621631112s) [noise=true , address=77.90.31.13:4658 , piece_id=DBR7S55I4MPSYYU6RZ53YPY7LRGOC7NLGJTPJN5WDKKSDFKK7XVQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.841007619 (1h44m20.621940997s) [noise=true , address=45.91.116.14:4194 , piece_id=ASNDK6UDUBYSWHRCAXXEZNDOOWHAVGTY4RLADLSBKOK6FQW7GL3Q, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.845339432 (1h44m20.626272821s) [seg_pos={0 0}, upload=1598]: Done begin upload segment. Jul 26 23:36:04.845652152 (1h44m20.626585541s) [noise=true , address=79.127.201.215:4358 , piece_id=A7C4IOLSJSOESPQH272ZD7Z4L4RRRTD73NFUPK427K35BXZQT4YQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.846510648 (1h44m20.627444036s) [noise=true , address=64.49.13.140:4508 , piece_id=NUTBHS4EYXX4CTBYXQO2FVICUEO4ZONBZORHKR5M7GYBMREAINKQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.851906006 (1h44m20.632839391s) [noise=true , address=79.127.220.98:4421 , piece_id=FA56MS4AYJR7BDEJYF6FRIY6EPRA3IBUXKBBN6BNW6GVTIFB4JWA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.852726707 (1h44m20.633660093s) [noise=true , address=79.127.205.235:4689 , piece_id=7OEOLJPAZX6RTIRRJW3HNBGG3CC6UCORPOGWSCBUZGM6AHA7BNXA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.854274177 (1h44m20.635207559s) [noise=true , address=77.90.31.18:4833 , piece_id=I2QO2PXWEEHCXK4AUSJRS74DI52ACO6SJZKN73TIX3DQYYQNN6PA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.859614926 (1h44m20.640548341s) [noise=true , address=79.127.219.38:4010 , piece_id=TXBGG5A5OU7M4DAGIONF3JD35M2GA4CKCOWY2LE6I4TT6DXGAE4A, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.860440553 (1h44m20.641373924s) [noise=true , address=104.36.82.46:4481 , piece_id=2RYU5YI4BCG5FUIDMQLOI4OAB3SJ2OPZ7OUTDISNWR3GXOW2GU5A, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.870288756 (1h44m20.651222136s) [noise=true , address=104.36.82.55:4224 , piece_id=ERBTARIXECO7WA5XV3UE7G7QMAKK6PVSZAX75SB73ZH6URX6IOYA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.870666345 (1h44m20.65159973s) [noise=true , address=79.127.205.246:4281 , piece_id=PDXNFEHEAI3TSG3Z5EXHZSK4ZQ672ISPX73WSD7AP4SM55NJFPRQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.875966232 (1h44m20.656899625s) [noise=true , address=104.36.82.47:4537 , piece_id=FJZIQOODCFHS7FIVA4ZKTE2HRMPGTP5MY257VV4CYJW4Y5JV65NQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.876709751 (1h44m20.65764314s) [noise=true , address=79.127.205.233:4646 , piece_id=QT2ZDPUOM4EHZNFBZLYP54ELXK6QMKAK2XJBRRA7R6GN5EK563ZA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.877549378 (1h44m20.658482795s) [noise=true , address=77.90.31.10:4535 , piece_id=L5O6NKB24SYXJ23URUTAFOV6YEPONAG4A2WFDXLY47N3TH2QNJKQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.878080592 (1h44m20.659013982s) [noise=true , address=45.91.116.9:4267 , piece_id=EY2KIBXWSQBVO73IS5WUVN3DKJ6XMKDU6KAYJRZJ7YGGZEY7M65Q, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.913897176 (1h44m20.69483056s) [noise=true , address=77.90.31.13:4658 , piece_id=DBR7S55I4MPSYYU6RZ53YPY7LRGOC7NLGJTPJN5WDKKSDFKK7XVQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: storage node overloaded, request limit: 2000 Jul 26 23:36:04.966654274 (1h44m20.747587659s) [upload=1598]: Next returned nil segment Jul 26 23:36:04.966677829 (1h44m20.747611219s) [upload=1598]: Waiting for error group managing segments... Jul 26 23:36:04.974998351 (1h44m20.755931733s) [noise=true , address=77.90.31.18:4833 , piece_id=I2QO2PXWEEHCXK4AUSJRS74DI52ACO6SJZKN73TIX3DQYYQNN6PA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:04.991088727 (1h44m20.772022089s) [noise=true , address=79.127.219.38:4010 , piece_id=TXBGG5A5OU7M4DAGIONF3JD35M2GA4CKCOWY2LE6I4TT6DXGAE4A, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.278501307 (1h44m21.059434686s) [noise=true , address=79.127.219.51:4196 , piece_id=U4NDAUEAXGWNBALDD32JBGNQLZQPE4T6IILT7JPWUE6WWNMYFBCQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.409195473 (1h44m21.190128838s) [noise=true , address=45.91.116.9:4267 , piece_id=EY2KIBXWSQBVO73IS5WUVN3DKJ6XMKDU6KAYJRZJ7YGGZEY7M65Q, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:36:05.458373441 (1h44m21.239306812s) [noise=true , address=79.127.205.235:4689 , piece_id=7OEOLJPAZX6RTIRRJW3HNBGG3CC6UCORPOGWSCBUZGM6AHA7BNXA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.479084903 (1h44m21.260018282s) [noise=true , address=79.127.205.246:4281 , piece_id=PDXNFEHEAI3TSG3Z5EXHZSK4ZQ672ISPX73WSD7AP4SM55NJFPRQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.499725574 (1h44m21.280658945s) [noise=true , address=79.127.205.233:4646 , piece_id=QT2ZDPUOM4EHZNFBZLYP54ELXK6QMKAK2XJBRRA7R6GN5EK563ZA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.522319306 (1h44m21.303252681s) [noise=true , address=79.127.205.245:4237 , piece_id=WGJX7TMGFWAS3ECOHZXNTNUKDXGDCYQX4RFJU2LTHHC3OE335DGQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:06.098427042 (1h44m21.879360425s) [noise=true , address=79.127.220.98:4421 , piece_id=FA56MS4AYJR7BDEJYF6FRIY6EPRA3IBUXKBBN6BNW6GVTIFB4JWA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:12.552777573 (1h44m28.333710944s) [noise=true , address=45.91.116.14:4194 , piece_id=ASNDK6UDUBYSWHRCAXXEZNDOOWHAVGTY4RLADLSBKOK6FQW7GL3Q, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:12.762711647 (1h44m28.54364501s) [noise=true , address=64.49.13.140:4508 , piece_id=NUTBHS4EYXX4CTBYXQO2FVICUEO4ZONBZORHKR5M7GYBMREAINKQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:28.232659699 (1h44m44.013593069s) [noise=true , address=79.127.201.215:4358 , piece_id=A7C4IOLSJSOESPQH272ZD7Z4L4RRRTD73NFUPK427K35BXZQT4YQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:40.636698729 (1h44m56.41763211s) [noise=true , address=77.90.31.10:4535 , piece_id=L5O6NKB24SYXJ23URUTAFOV6YEPONAG4A2WFDXLY47N3TH2QNJKQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:37:04.754269978 (1h45m20.535203364s) [noise=true , address=104.36.82.46:4481 , piece_id=2RYU5YI4BCG5FUIDMQLOI4OAB3SJ2OPZ7OUTDISNWR3GXOW2GU5A, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 159.223.143.39:55094->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754421221 (1h45m20.535354657s) [noise=true , address=104.36.82.55:4224 , piece_id=ERBTARIXECO7WA5XV3UE7G7QMAKK6PVSZAX75SB73ZH6URX6IOYA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: rpc: dial tcp 104.36.82.55:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754480208 (1h45m20.535413592s) [noise=true , address=104.36.82.54:4187 , piece_id=BIWDUNNMQZGGWA7P3HOHLVEWC3LY7UUWORMY6LIOK7UN55BA55BQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: rpc: dial tcp 104.36.82.54:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754510791 (1h45m20.535444158s) [noise=true , address=104.36.82.47:4537 , piece_id=FJZIQOODCFHS7FIVA4ZKTE2HRMPGTP5MY257VV4CYJW4Y5JV65NQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: write tcp 159.223.143.39:50500->104.36.82.47:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754561974 (1h45m20.535495355s) [seg_pos={0 0}, upload=1598]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:37:04.787407176 (1h45m20.56834056s) [upload=1598]: Done uploading segments... Jul 26 23:37:04.787439236 (1h45m20.568372624s) [upload=1598]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.035061349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X..........XX...X...X..XX.X...X..X.......X...X....XX.......] duration: 15.133966245s FAIL (2025-07-26 23:36:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.13399621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.X..........XX...X...X..XX.X...X..X.......X...X....XX......] duration: 48.392081559s FAIL (2025-07-26 23:36:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.392151508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.038290865s FAIL (2025-07-26 23:36:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.03830913s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031293562s FAIL (2025-07-26 23:36:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:40800->104.36.82.9:4007: use of closed network connection; conn closed: op Jul 26 23:35:10.187069630 (1h43m25.968003081s) [seg_pos={0 0}, upload=1577]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:35:10.229261143 (1h43m26.010194523s) [upload=1577]: Done uploading segments... Jul 26 23:35:10.229291203 (1h43m26.010224581s) [upload=1577]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 23:35:53.945803521 (1h44m9.7267369s) [upload=1595]: Starting upload Jul 26 23:35:53.945890043 (1h44m9.726823432s) [upload=1595]: Done starting upload Jul 26 23:35:53.946079246 (1h44m9.727012614s) [upload=1595]: Uploading segments... Jul 26 23:35:53.946242699 (1h44m9.727176084s) [upload=1595]: Got next segment. Inline: false Jul 26 23:35:53.946317828 (1h44m9.727251214s) [upload=1595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:35:53.946324734 (1h44m9.727258113s) [upload=1595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:35:54.002502015 (1h44m9.783435432s) [seg_pos={0 0}, upload=1595]: Begin upload segment... Jul 26 23:35:54.002564052 (1h44m9.783497421s) [seg_pos={0 0}, upload=1595]: Done begin upload segment. Jul 26 23:35:54.002614701 (1h44m9.783548092s) [noise=true , address=104.36.82.7:4910 , piece_id=KNJD4UFEQIZFVP3FVJS4VTFVCYDKTHWCP6AO7LM3Y4VD4BJVD7WA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.003215323 (1h44m9.784148736s) [noise=true , address=104.36.82.15:4259 , piece_id=QONEJXNC5ONNMACLGKRWD5X74BIACXZHD7LV4U3YOAL6CN4RX3SA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.003659788 (1h44m9.784593179s) [noise=true , address=45.91.116.11:4117 , piece_id=6EZHL64PLCD6XYMA54ASRGFKKJCNYDIIJ6V4RZIYUP2FIGRAOSGA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.004215855 (1h44m9.785149244s) [noise=true , address=104.36.82.5:4837 , piece_id=NMFZCJDLSMSZBEV73YRPYKNKNPC4MBJM2QHFPBHI72ISCXDJDVAA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.004606430 (1h44m9.785539816s) [noise=true , address=77.90.31.21:4879 , piece_id=ADR6FOQ4TENZFJH5KQNSLHWUSDNIK54JF2R3BSU5H76HSPNL7E4A, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.005019070 (1h44m9.785952463s) [noise=true , address=104.36.82.10:4078 , piece_id=6XMEP5XDE2425LNJA7XH6YKHOXYUMWVY3TCBYIWQYXKRW2LU24SA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.005419168 (1h44m9.786352558s) [noise=true , address=45.91.116.7:4009 , piece_id=72GEVH5X23BLLRCARPR6XPVA4OFW24USCTUNDZ6OYRWFWGYA4KZQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.005753828 (1h44m9.786687206s) [noise=true , address=45.91.116.12:4227 , piece_id=2NLQHWHBYHOP5CCKSSIC46KZGNPWU4GZWETDPGYM2DQJORPQLJTQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.006045314 (1h44m9.786978693s) [noise=true , address=104.36.82.4:4804 , piece_id=GFZJIBUTQREFDFFCRO2U7TD3Q7UMZTZSGFGUQ33KSGXYQO7BZMPQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.006383534 (1h44m9.78731694s) [noise=true , address=45.91.116.8:4048 , piece_id=4XICQALXI5HDIMIQOPHN4QOATOLPV25QOPPCXFAQ2VOOZP523Y2Q, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.006788825 (1h44m9.787722207s) [noise=true , address=104.36.82.9:4007 , piece_id=F7H5YO4JZIZUE2W7FMYXAVCLBFHAXVGUHXC33BGZZRLAXDHRTCEQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.007315997 (1h44m9.788249396s) [noise=true , address=77.90.31.14:4661 , piece_id=DVOKWGSMWC3VQSICELUO24C2EA2ZYBB34FFO4RYD4L2PLOO5ARZA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.007740060 (1h44m9.788673464s) [noise=true , address=77.90.31.13:4628 , piece_id=BIRTXQQST3KOOXMJL2GQM2ZJNXDK2QEG67DUPLEXHOERLOWC4IVQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.008127233 (1h44m9.789060639s) [noise=true , address=45.91.116.10:4083 , piece_id=6N6KEG3X3ADBK6RV34RJQXQPTLUWO752X2B3J6P2SNJ2PMR3SZNA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.008471373 (1h44m9.789404759s) [noise=true , address=77.90.31.14:4663 , piece_id=6AW2IWP7C3MM7WJGNY45IPIGTPLKXAXO6P2FX43YZIOZWZDZZO2Q, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.008815453 (1h44m9.789748835s) [noise=true , address=45.91.116.6:4974 , piece_id=BJNYPEDFQOLVAX2Z2W7V7NBQSPLZ73K7F5G5EE7OAHUT2P6W3APA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.143934908 (1h44m9.924868276s) [upload=1595]: Next returned nil segment Jul 26 23:35:54.144039942 (1h44m9.924973323s) [upload=1595]: Waiting for error group managing segments... Jul 26 23:35:54.148543670 (1h44m9.929477041s) [noise=true , address=45.91.116.8:4048 , piece_id=4XICQALXI5HDIMIQOPHN4QOATOLPV25QOPPCXFAQ2VOOZP523Y2Q, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.149295481 (1h44m9.930228852s) [noise=true , address=45.91.116.6:4974 , piece_id=BJNYPEDFQOLVAX2Z2W7V7NBQSPLZ73K7F5G5EE7OAHUT2P6W3APA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.149820690 (1h44m9.930754083s) [noise=true , address=45.91.116.12:4227 , piece_id=2NLQHWHBYHOP5CCKSSIC46KZGNPWU4GZWETDPGYM2DQJORPQLJTQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.150456218 (1h44m9.931389581s) [noise=true , address=45.91.116.10:4083 , piece_id=6N6KEG3X3ADBK6RV34RJQXQPTLUWO752X2B3J6P2SNJ2PMR3SZNA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.151182076 (1h44m9.932115454s) [noise=true , address=77.90.31.14:4663 , piece_id=6AW2IWP7C3MM7WJGNY45IPIGTPLKXAXO6P2FX43YZIOZWZDZZO2Q, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.151697969 (1h44m9.932631351s) [noise=true , address=45.91.116.7:4009 , piece_id=72GEVH5X23BLLRCARPR6XPVA4OFW24USCTUNDZ6OYRWFWGYA4KZQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.152298868 (1h44m9.933232249s) [noise=true , address=77.90.31.21:4879 , piece_id=ADR6FOQ4TENZFJH5KQNSLHWUSDNIK54JF2R3BSU5H76HSPNL7E4A, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.154011055 (1h44m9.934944425s) [noise=true , address=77.90.31.14:4661 , piece_id=DVOKWGSMWC3VQSICELUO24C2EA2ZYBB34FFO4RYD4L2PLOO5ARZA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.280558511 (1h44m10.061491902s) [noise=true , address=45.91.116.11:4117 , piece_id=6EZHL64PLCD6XYMA54ASRGFKKJCNYDIIJ6V4RZIYUP2FIGRAOSGA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:55.541743144 (1h44m11.322676507s) [noise=true , address=77.90.31.13:4628 , piece_id=BIRTXQQST3KOOXMJL2GQM2ZJNXDK2QEG67DUPLEXHOERLOWC4IVQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:36:23.913398522 (1h44m39.69433191s) [noise=true , address=104.36.82.5:4837 , piece_id=NMFZCJDLSMSZBEV73YRPYKNKNPC4MBJM2QHFPBHI72ISCXDJDVAA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.913804046 (1h44m39.694737422s) [noise=true , address=104.36.82.4:4804 , piece_id=GFZJIBUTQREFDFFCRO2U7TD3Q7UMZTZSGFGUQ33KSGXYQO7BZMPQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.913915334 (1h44m39.694848725s) [noise=true , address=104.36.82.7:4910 , piece_id=KNJD4UFEQIZFVP3FVJS4VTFVCYDKTHWCP6AO7LM3Y4VD4BJVD7WA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:35000->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914111654 (1h44m39.695045053s) [noise=true , address=104.36.82.9:4007 , piece_id=F7H5YO4JZIZUE2W7FMYXAVCLBFHAXVGUHXC33BGZZRLAXDHRTCEQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; context deadline exceeded Jul 26 23:36:23.914461411 (1h44m39.695394807s) [noise=true , address=104.36.82.15:4259 , piece_id=QONEJXNC5ONNMACLGKRWD5X74BIACXZHD7LV4U3YOAL6CN4RX3SA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:42280->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914522870 (1h44m39.695456249s) [noise=true , address=104.36.82.10:4078 , piece_id=6XMEP5XDE2425LNJA7XH6YKHOXYUMWVY3TCBYIWQYXKRW2LU24SA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:56724->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914569057 (1h44m39.69550244s) [seg_pos={0 0}, upload=1595]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:36:23.942316640 (1h44m39.723250007s) [upload=1595]: Done uploading segments... Jul 26 23:36:23.942360746 (1h44m39.723294114s) [upload=1595]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.031324218s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....................XXX.X....XX.X.X.X.......XXX...XX.....X.] duration: 793.451722ms FAIL (2025-07-26 23:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 793.511098ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......................XXX.X....XX.X.X.X.......XXX...XX.....X] duration: 30.001191982s FAIL (2025-07-26 23:36:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001217724s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.038812307s FAIL (2025-07-26 23:36:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.038923968s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [X.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03823959s FAIL (2025-07-26 23:36:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== h43m16.754324307s) [noise=true , address=104.36.82.52:4110 , piece_id=24FA2HLMLASP6WPAR56VNXLVL5IDUMPB6ZZIP4TMBEJTYQ4HCNNA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.973825790 (1h43m16.754759175s) [noise=true , address=77.90.31.19:4871 , piece_id=OFXW3YMPPV2ZIRYU56ZBIC6SBZZE4VSVDVIZ6EGYX3JLZDFZV35Q, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.974224488 (1h43m16.755157868s) [noise=true , address=79.127.205.246:4283 , piece_id=3YOJN4D4BJORXJSGKMCSZRH4LJNQTUTVLF3DXYW56DL6N7B6EXWQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.974555382 (1h43m16.755488761s) [noise=true , address=79.127.219.47:4048 , piece_id=FNS2XCT3IO3RARX6BVSNXHIYJG52WOO3GU3EINVIRUNE55WMWUNA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.974906971 (1h43m16.75584035s) [noise=true , address=79.127.205.235:4666 , piece_id=F5CXC2EPUEY3TP7MNZ6GQZLEBTWRSQ2336V5ADAFV57KPXEJF72A, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.975327407 (1h43m16.75626092s) [noise=true , address=79.127.205.234:4610 , piece_id=5Q7E7MSV4U2XGOH2PBIEMWCTCATJFPO44UAASMX6KDFN6GNI2K2Q, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.975653416 (1h43m16.756586796s) [noise=true , address=104.36.82.55:4228 , piece_id=4E4ST7GBTNSANU4Z4FREQWJ4XT4GDTPLUBGJNYL2YLTE3KF6RO4A, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.976007834 (1h43m16.756941214s) [noise=true , address=104.36.82.53:4129 , piece_id=FESURDBXTPBQXCKF74GMGDXKBLKQL7PUW4KMVQMXBT2BXN2ARJIQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.976511755 (1h43m16.757445144s) [noise=true , address=104.36.82.47:4519 , piece_id=EV6VCGCB6BXPRVSJ6YZP2RI4ZMZ253QL2QQLBQQW7VE7K5NKWWFQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.976936284 (1h43m16.757869691s) [noise=true , address=79.127.219.38:4013 , piece_id=XGZ5CUTW2CEUO4KB47Q7QT4V2VL64N5FDKVTYDQGIVE4W3RD3OZQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.977330624 (1h43m16.75826401s) [noise=true , address=64.49.13.139:4478 , piece_id=OLE5U7SOON2TSRHXNBWXQ42WMM5K3W2KP5C5I54P2KLT3KDITCDA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.977677177 (1h43m16.758610566s) [noise=true , address=104.36.82.56:4248 , piece_id=U4XDOZ3WJSTB4FPFAANSNPS4OOVS7U6PXHUM54Q26XOGT3XV7VTQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.978038406 (1h43m16.758971796s) [noise=true , address=79.127.220.101:4048 , piece_id=ZHW4564FCPGILAJNFPQ2TMFMNEU44ECWGISTRXO4PUJGI5WRCWWA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.978415254 (1h43m16.759348637s) [noise=true , address=77.90.31.17:4784 , piece_id=6BWLTCKJKON6LCHS26WXHUZSVEYCK7FKMGQS6MAOGWPUCNAMSVVA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.978745320 (1h43m16.7596787s) [noise=true , address=104.36.82.54:4177 , piece_id=77TVG6CBPKRWOBWUZY2OXXIK6IJRRUPLIGEWPAOK7GW4C7LCA3PA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.979171594 (1h43m16.760104973s) [noise=true , address=104.36.82.46:4507 , piece_id=MJU7UMBGJ5MYY22S2ITFNAC36VHTLEDJDGRGM7W5BBX3NUR4DZOQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.979533662 (1h43m16.760467062s) [noise=true , address=77.90.31.15:4703 , piece_id=DA262R7EE6WYT4OL4IQ2IZ3OOTXY65NJRI4JIFFGKZQ4XMYVLBOA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.979876289 (1h43m16.760809668s) [noise=true , address=79.127.205.245:4249 , piece_id=NB4FR2XNBFH3Z3BVBJALBFA36OKL4XSG5YOCN7IXZ5FBNMC2UVHA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:01.107362375 (1h43m16.888295744s) [upload=1584]: Next returned nil segment Jul 26 23:35:01.107391491 (1h43m16.888324857s) [upload=1584]: Waiting for error group managing segments... Jul 26 23:35:01.134793692 (1h43m16.915727069s) [noise=true , address=77.90.31.15:4703 , piece_id=DA262R7EE6WYT4OL4IQ2IZ3OOTXY65NJRI4JIFFGKZQ4XMYVLBOA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.188989560 (1h43m16.969922924s) [noise=true , address=79.127.219.38:4013 , piece_id=XGZ5CUTW2CEUO4KB47Q7QT4V2VL64N5FDKVTYDQGIVE4W3RD3OZQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.697000738 (1h43m17.477934124s) [noise=true , address=79.127.205.234:4610 , piece_id=5Q7E7MSV4U2XGOH2PBIEMWCTCATJFPO44UAASMX6KDFN6GNI2K2Q, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.729387614 (1h43m17.510320986s) [noise=true , address=79.127.205.235:4666 , piece_id=F5CXC2EPUEY3TP7MNZ6GQZLEBTWRSQ2336V5ADAFV57KPXEJF72A, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.818316840 (1h43m17.599250222s) [noise=true , address=79.127.205.246:4283 , piece_id=3YOJN4D4BJORXJSGKMCSZRH4LJNQTUTVLF3DXYW56DL6N7B6EXWQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.885206499 (1h43m17.666139885s) [noise=true , address=79.127.205.245:4249 , piece_id=NB4FR2XNBFH3Z3BVBJALBFA36OKL4XSG5YOCN7IXZ5FBNMC2UVHA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:02.628468909 (1h43m18.409402291s) [noise=true , address=64.49.13.139:4478 , piece_id=OLE5U7SOON2TSRHXNBWXQ42WMM5K3W2KP5C5I54P2KLT3KDITCDA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:05.099767376 (1h43m20.880700741s) [noise=true , address=79.127.219.47:4048 , piece_id=FNS2XCT3IO3RARX6BVSNXHIYJG52WOO3GU3EINVIRUNE55WMWUNA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:05.400170017 (1h43m21.181103382s) [noise=true , address=79.127.220.101:4048 , piece_id=ZHW4564FCPGILAJNFPQ2TMFMNEU44ECWGISTRXO4PUJGI5WRCWWA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:05.877677709 (1h43m21.658611084s) [noise=true , address=77.90.31.19:4871 , piece_id=OFXW3YMPPV2ZIRYU56ZBIC6SBZZE4VSVDVIZ6EGYX3JLZDFZV35Q, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:36:00.810037130 (1h44m16.590970511s) [noise=true , address=104.36.82.52:4110 , piece_id=24FA2HLMLASP6WPAR56VNXLVL5IDUMPB6ZZIP4TMBEJTYQ4HCNNA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810135931 (1h44m16.59106931s) [noise=true , address=104.36.82.55:4228 , piece_id=4E4ST7GBTNSANU4Z4FREQWJ4XT4GDTPLUBGJNYL2YLTE3KF6RO4A, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 159.223.143.39:56682->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810249290 (1h44m16.591182673s) [noise=true , address=104.36.82.53:4129 , piece_id=FESURDBXTPBQXCKF74GMGDXKBLKQL7PUW4KMVQMXBT2BXN2ARJIQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810372132 (1h44m16.591305511s) [noise=true , address=104.36.82.56:4248 , piece_id=U4XDOZ3WJSTB4FPFAANSNPS4OOVS7U6PXHUM54Q26XOGT3XV7VTQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: rpc: dial tcp 104.36.82.56:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810432825 (1h44m16.591366207s) [noise=true , address=77.90.31.17:4784 , piece_id=6BWLTCKJKON6LCHS26WXHUZSVEYCK7FKMGQS6MAOGWPUCNAMSVVA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: rpc: dial tcp 77.90.31.17:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810495712 (1h44m16.591429095s) [noise=true , address=104.36.82.47:4519 , piece_id=EV6VCGCB6BXPRVSJ6YZP2RI4ZMZ253QL2QQLBQQW7VE7K5NKWWFQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: rpc: dial tcp 104.36.82.47:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810502307 (1h44m16.591435687s) [noise=true , address=104.36.82.54:4177 , piece_id=77TVG6CBPKRWOBWUZY2OXXIK6IJRRUPLIGEWPAOK7GW4C7LCA3PA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 159.223.143.39:59922->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810575263 (1h44m16.591508646s) [noise=true , address=104.36.82.46:4507 , piece_id=MJU7UMBGJ5MYY22S2ITFNAC36VHTLEDJDGRGM7W5BBX3NUR4DZOQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: write tcp 159.223.143.39:48250->104.36.82.46:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810607709 (1h44m16.591541099s) [seg_pos={0 0}, upload=1584]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:36:00.844247378 (1h44m16.625180761s) [upload=1584]: Done uploading segments... Jul 26 23:36:00.844303507 (1h44m16.625236887s) [upload=1584]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.038260719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X.X..........XX...X...X..XX.X...X..X.......X...X....XX.....] duration: 53.960524941s FAIL (2025-07-26 23:35:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.960549432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [..................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.040356041s FAIL (2025-07-26 23:35:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.040378536s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.045336327s FAIL (2025-07-26 23:35:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057416370 (1h42m55.838349747s) [seg_pos={0 0}, upload=1569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:34:40.082099388 (1h42m55.863032755s) [upload=1569]: Done uploading segments... Jul 26 23:34:40.082136123 (1h42m55.863069512s) [upload=1569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:34:40.221455653 (1h42m56.002389043s) [upload=1577]: Starting upload Jul 26 23:34:40.221529861 (1h42m56.002463241s) [upload=1577]: Done starting upload Jul 26 23:34:40.221907241 (1h42m56.002840619s) [upload=1577]: Uploading segments... Jul 26 23:34:40.221972845 (1h42m56.002906232s) [upload=1577]: Got next segment. Inline: false Jul 26 23:34:40.222014586 (1h42m56.002947972s) [upload=1577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:34:40.222021086 (1h42m56.002954471s) [upload=1577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:34:40.252197297 (1h42m56.033130666s) [seg_pos={0 0}, upload=1577]: Begin upload segment... Jul 26 23:34:40.252246145 (1h42m56.033179509s) [seg_pos={0 0}, upload=1577]: Done begin upload segment. Jul 26 23:34:40.252280488 (1h42m56.033213855s) [noise=true , address=104.36.82.7:4912 , piece_id=T7YQQSISDNHOMQLFBBAHEQLQGSW523PR6RQQFO4QDWALMU2RUCUA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.252805131 (1h42m56.033738511s) [noise=true , address=77.90.31.16:4735 , piece_id=D5KXRRKTLUNUEL4FJPF4MQKQURHFUTBTEEFTEIBX3CIL3PSCIJ3A, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253198175 (1h42m56.034131559s) [noise=true , address=104.36.82.9:4007 , piece_id=GLUQRUTMINCAGR6UG7FANYL6RFTQWTP5JBEN5C5CUCYJQQOKDYQQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253266603 (1h42m56.034199985s) [noise=true , address=45.91.116.14:4191 , piece_id=ODGPV6SB5JGIDBYGBZUSPNDT5DTKOVUYZPEDF6DI7ZNXEGJOJTAA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253599594 (1h42m56.034532985s) [noise=true , address=45.91.116.7:4009 , piece_id=XOFH5XKVBS2NPEMNHUGGGC5KOTV3THQCVNZ5XWAMK7JJL3AFKRQQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253849541 (1h42m56.034782929s) [noise=true , address=45.91.116.13:4155 , piece_id=QGX3TNGBFMQ5AMYWADEAW5QN6HUM5CAGBPINAMIBLJICYPNHCVKA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253990096 (1h42m56.034923483s) [noise=true , address=45.91.116.8:4048 , piece_id=ODMOI2FGYQSW5X6BMPD7DU2HH7FBYTUXEABPCQLHWBSDLLFFQOUA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.254389355 (1h42m56.035322743s) [noise=true , address=45.91.116.6:4975 , piece_id=66JBELL73G6MZX6W7MRXJJOHMMKBOCBESMJFCAG5PSCCU2MHAGRA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.255161978 (1h42m56.036095364s) [noise=true , address=45.91.116.8:4047 , piece_id=DGCAIYY7ETLGSDK3RP4FCOTFKP4UBC5DN4OOMVO7SIE5V6QRW7DQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.259405366 (1h42m56.040338757s) [noise=true , address=45.91.116.12:4226 , piece_id=M3O4ZJXNS6LU7W6SGXRHDNCRLCQ2DBPQZX2W4UKJUVI6K6DZXK5A, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.259926050 (1h42m56.04085944s) [noise=true , address=104.36.82.11:4114 , piece_id=EOX3AHREXUGSRRNKRSE7VUTVULVDFL2E6OUJNDJP2L2FTJT7CFOA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.260475877 (1h42m56.041409267s) [noise=true , address=104.36.82.13:4187 , piece_id=S2CONJADAQA3KKKF2AOECWFVVQXUL3SLCSGBQCAODKORKMTZFAEQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.260875663 (1h42m56.041809049s) [noise=true , address=45.91.116.11:4119 , piece_id=OEMB2DVQUTMR2WLPUSNLVX3JDJGD2PFEGHE3NERKLMECKFUFQUAQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.261288596 (1h42m56.042221988s) [noise=true , address=104.36.82.5:4839 , piece_id=GKTRQ5HNYVZX2UGEIMIUQZQWUYNV7UYKMTXWSMAFB65R532Z7ASQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.262068256 (1h42m56.043001646s) [noise=true , address=104.36.82.12:4150 , piece_id=HTEXFXTBOWURKZMALKP6WXYBPYNWCV25VENYNJ5JENMTJPQXYJRA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.263574098 (1h42m56.044507488s) [noise=true , address=45.91.116.6:4973 , piece_id=5J3PWP6FDMMY4J23Q7UBSZ6C4QERKYAWH7N73VSMAEGLDYIPZZHQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.363753637 (1h42m56.14468702s) [upload=1577]: Next returned nil segment Jul 26 23:34:40.363776740 (1h42m56.144710127s) [upload=1577]: Waiting for error group managing segments... Jul 26 23:34:40.369622990 (1h42m56.150556361s) [noise=true , address=45.91.116.12:4226 , piece_id=M3O4ZJXNS6LU7W6SGXRHDNCRLCQ2DBPQZX2W4UKJUVI6K6DZXK5A, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370205840 (1h42m56.151139227s) [noise=true , address=45.91.116.8:4047 , piece_id=DGCAIYY7ETLGSDK3RP4FCOTFKP4UBC5DN4OOMVO7SIE5V6QRW7DQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370231866 (1h42m56.151165304s) [noise=true , address=45.91.116.7:4009 , piece_id=XOFH5XKVBS2NPEMNHUGGGC5KOTV3THQCVNZ5XWAMK7JJL3AFKRQQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370695637 (1h42m56.151629002s) [noise=true , address=45.91.116.13:4155 , piece_id=QGX3TNGBFMQ5AMYWADEAW5QN6HUM5CAGBPINAMIBLJICYPNHCVKA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370790748 (1h42m56.151724118s) [noise=true , address=45.91.116.8:4048 , piece_id=ODMOI2FGYQSW5X6BMPD7DU2HH7FBYTUXEABPCQLHWBSDLLFFQOUA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370835810 (1h42m56.15176918s) [noise=true , address=45.91.116.6:4973 , piece_id=5J3PWP6FDMMY4J23Q7UBSZ6C4QERKYAWH7N73VSMAEGLDYIPZZHQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.371122339 (1h42m56.152055708s) [noise=true , address=45.91.116.6:4975 , piece_id=66JBELL73G6MZX6W7MRXJJOHMMKBOCBESMJFCAG5PSCCU2MHAGRA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.372951176 (1h42m56.15388455s) [noise=true , address=45.91.116.11:4119 , piece_id=OEMB2DVQUTMR2WLPUSNLVX3JDJGD2PFEGHE3NERKLMECKFUFQUAQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.409541510 (1h42m56.19047488s) [noise=true , address=77.90.31.16:4735 , piece_id=D5KXRRKTLUNUEL4FJPF4MQKQURHFUTBTEEFTEIBX3CIL3PSCIJ3A, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.552342650 (1h42m56.333276016s) [noise=true , address=45.91.116.14:4191 , piece_id=ODGPV6SB5JGIDBYGBZUSPNDT5DTKOVUYZPEDF6DI7ZNXEGJOJTAA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:35:10.186470134 (1h43m25.967403516s) [noise=true , address=104.36.82.11:4114 , piece_id=EOX3AHREXUGSRRNKRSE7VUTVULVDFL2E6OUJNDJP2L2FTJT7CFOA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186562656 (1h43m25.967496026s) [noise=true , address=104.36.82.7:4912 , piece_id=T7YQQSISDNHOMQLFBBAHEQLQGSW523PR6RQQFO4QDWALMU2RUCUA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:33556->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186614272 (1h43m25.967547664s) [noise=true , address=104.36.82.13:4187 , piece_id=S2CONJADAQA3KKKF2AOECWFVVQXUL3SLCSGBQCAODKORKMTZFAEQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:35:10.186743170 (1h43m25.967676553s) [noise=true , address=104.36.82.5:4839 , piece_id=GKTRQ5HNYVZX2UGEIMIUQZQWUYNV7UYKMTXWSMAFB65R532Z7ASQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:49742->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186864056 (1h43m25.967797426s) [noise=true , address=104.36.82.12:4150 , piece_id=HTEXFXTBOWURKZMALKP6WXYBPYNWCV25VENYNJ5JENMTJPQXYJRA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:39734->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186952833 (1h43m25.967886198s) [noise=true , address=104.36.82.9:4007 , piece_id=GLUQRUTMINCAGR6UG7FANYL6RFTQWTP5JBEN5C5CUCYJQQOKDYQQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:40800->104.36.82.9:4007: use of closed network connection; conn closed: op Jul 26 23:35:10.187069630 (1h43m25.968003081s) [seg_pos={0 0}, upload=1577]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:35:10.229261143 (1h43m26.010194523s) [upload=1577]: Done uploading segments... Jul 26 23:35:10.229291203 (1h43m26.010224581s) [upload=1577]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.045359687s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................XXX.X....XX.X.X.X.......XXX...XX.....] duration: 2.023487935s FAIL (2025-07-26 23:35:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.02351824s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.048912424s FAIL (2025-07-26 23:35:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== equestItem_SegmentBegin Jul 26 23:34:00.737700749 (1h42m16.518634134s) [seg_pos={0 0}, upload=1568]: Begin upload segment... Jul 26 23:34:00.737772247 (1h42m16.518705616s) [seg_pos={0 0}, upload=1568]: Done begin upload segment. Jul 26 23:34:00.737799270 (1h42m16.518732657s) [noise=true , address=104.36.82.47:4523 , piece_id=CTYLPPTWOPDNEBFNBT62ADKLLOB4A7DUKARJEVDGTLJ3WTQJMCIQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.737819822 (1h42m16.518753317s) [noise=true , address=104.36.82.54:4166 , piece_id=PDKUUD2O4C6G4NOCZGKI5CDD6GFWCHJISJSYEA7CYI22DIZEGSUQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.738347939 (1h42m16.519281432s) [noise=true , address=45.91.116.9:4270 , piece_id=GS52D7IGNNUSLOMMMW7VWRRXQBWF6IHHD5JRIZZ4TL35RCOUTT2A, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.738429916 (1h42m16.519363288s) [noise=true , address=79.127.201.214:4289 , piece_id=ILR652WR4DVVPDBQWK3GSMZPAQQSLXXXBTNEW3MHJZW5CRTZ46NQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.738833033 (1h42m16.519766418s) [noise=true , address=77.90.31.17:4782 , piece_id=AGSEE4JIZCCOCCLJLADMRX3TPVG4ZFCKCNF6QEZLE5G2JO36OT7Q, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.740352024 (1h42m16.521285392s) [noise=true , address=79.127.220.101:4041 , piece_id=HPTWDVEQTZDQG6OGPTR54LQ56VAHJGRU3LGPUQHDJJ3U4HFB35FA, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.740773144 (1h42m16.521706533s) [noise=true , address=156.146.43.225:4434 , piece_id=5VMHHR3IMZBDUJAGOVTHZXKKN6J56P76QMTLRTV3XVYY2BYE5QIQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.740867890 (1h42m16.521801272s) [noise=true , address=79.127.219.38:4001 , piece_id=4VMGEJ4HSUNWOMS6GVGR7MRET42PW75BRQRFYJWNB6E5ZCMUYMGQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.741303389 (1h42m16.522236771s) [noise=true , address=79.127.220.98:4422 , piece_id=66OIKBF2AM7JBWSFSMFRYUU7BGMAMBQQ7KIWCN4AUOU4MVVAUQXA, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.741787134 (1h42m16.522720516s) [noise=true , address=104.36.82.46:4501 , piece_id=3FSD3WE32RQG6OAXVSL2YV2CH2NGUVMCWKKR4JU3JZNTOKCSZDUQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.742286052 (1h42m16.523219439s) [noise=true , address=104.36.82.56:4263 , piece_id=X6HWOLT7LHT2ME2LBS7GPX7WPBVWPZTKOJQP7E54TJ2HWHDZPCJQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.742721148 (1h42m16.52365453s) [noise=true , address=79.127.205.245:4245 , piece_id=KH3CAFF2WWLJDX3QZEK7XKOF4CRHU6LS73KJZVJNEUQPZG2YWVXA, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.743436565 (1h42m16.524369952s) [noise=true , address=79.127.205.246:4282 , piece_id=6NQRVHEYIKJDWKL6J3KU5X2Y6IPFBS6ARSM6RY4X672PXOHMZBYQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.744077419 (1h42m16.525010833s) [noise=true , address=79.127.201.215:4354 , piece_id=U5BRN732WWNRCXELQA25AGCO4VPKYSTMNZUJMUPS36YMJ6IISVFQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.744526597 (1h42m16.525459988s) [noise=true , address=79.127.205.234:4607 , piece_id=BTFFNZBUQ5AQRL35YXMW3P53D6AAYRB27DWXJ5ELPKPYK7QX2PGQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.745079232 (1h42m16.526012708s) [noise=true , address=104.36.82.55:4207 , piece_id=QV35AAUSONDCSLM2P64N7D5ZHPQHPTXAKHPTQCGTVAOXT7HG2V4Q, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.745284785 (1h42m16.52621817s) [noise=true , address=104.36.82.52:4106 , piece_id=3WNFALTDWUD2VELI3XJKOHZ44OSZZPBHIWWV3DLYHJ77UQCU3XIQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.745649491 (1h42m16.526582871s) [noise=true , address=64.49.13.138:4482 , piece_id=L7LZREKAH3SSNUZR7JXA63OVVLMQ3HW72QVIQYAPO3LQYFO6ZRPQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.850787830 (1h42m16.631721196s) [upload=1568]: Next returned nil segment Jul 26 23:34:00.850811119 (1h42m16.631744483s) [upload=1568]: Waiting for error group managing segments... Jul 26 23:34:00.856897735 (1h42m16.637831112s) [noise=true , address=79.127.219.38:4001 , piece_id=4VMGEJ4HSUNWOMS6GVGR7MRET42PW75BRQRFYJWNB6E5ZCMUYMGQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.016511260 (1h42m16.797444629s) [noise=true , address=79.127.201.214:4289 , piece_id=ILR652WR4DVVPDBQWK3GSMZPAQQSLXXXBTNEW3MHJZW5CRTZ46NQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.333563976 (1h42m17.114497372s) [noise=true , address=79.127.205.234:4607 , piece_id=BTFFNZBUQ5AQRL35YXMW3P53D6AAYRB27DWXJ5ELPKPYK7QX2PGQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.347878694 (1h42m17.128812063s) [noise=true , address=79.127.205.245:4245 , piece_id=KH3CAFF2WWLJDX3QZEK7XKOF4CRHU6LS73KJZVJNEUQPZG2YWVXA, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.412325845 (1h42m17.193259227s) [noise=true , address=79.127.205.246:4282 , piece_id=6NQRVHEYIKJDWKL6J3KU5X2Y6IPFBS6ARSM6RY4X672PXOHMZBYQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.432330393 (1h42m17.213263775s) [noise=true , address=64.49.13.138:4482 , piece_id=L7LZREKAH3SSNUZR7JXA63OVVLMQ3HW72QVIQYAPO3LQYFO6ZRPQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:02.342277081 (1h42m18.123210452s) [noise=true , address=156.146.43.225:4434 , piece_id=5VMHHR3IMZBDUJAGOVTHZXKKN6J56P76QMTLRTV3XVYY2BYE5QIQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:03.623276829 (1h42m19.4042102s) [noise=true , address=79.127.220.101:4041 , piece_id=HPTWDVEQTZDQG6OGPTR54LQ56VAHJGRU3LGPUQHDJJ3U4HFB35FA, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:04.134317249 (1h42m19.915250632s) [noise=true , address=79.127.220.98:4422 , piece_id=66OIKBF2AM7JBWSFSMFRYUU7BGMAMBQQ7KIWCN4AUOU4MVVAUQXA, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:06.500220285 (1h42m22.281153655s) [noise=true , address=45.91.116.9:4270 , piece_id=GS52D7IGNNUSLOMMMW7VWRRXQBWF6IHHD5JRIZZ4TL35RCOUTT2A, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:43.282849015 (1h42m59.063782391s) [noise=true , address=79.127.201.215:4354 , piece_id=U5BRN732WWNRCXELQA25AGCO4VPKYSTMNZUJMUPS36YMJ6IISVFQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:35:00.662109754 (1h43m16.443043122s) [noise=true , address=104.36.82.46:4501 , piece_id=3FSD3WE32RQG6OAXVSL2YV2CH2NGUVMCWKKR4JU3JZNTOKCSZDUQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: rpc: dial tcp 104.36.82.46:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.662934418 (1h43m16.443867799s) [noise=true , address=77.90.31.17:4782 , piece_id=AGSEE4JIZCCOCCLJLADMRX3TPVG4ZFCKCNF6QEZLE5G2JO36OT7Q, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: rpc: dial tcp 77.90.31.17:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663110677 (1h43m16.444044057s) [noise=true , address=104.36.82.56:4263 , piece_id=X6HWOLT7LHT2ME2LBS7GPX7WPBVWPZTKOJQP7E54TJ2HWHDZPCJQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: rpc: dial tcp 104.36.82.56:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663225896 (1h43m16.444159278s) [noise=true , address=104.36.82.55:4207 , piece_id=QV35AAUSONDCSLM2P64N7D5ZHPQHPTXAKHPTQCGTVAOXT7HG2V4Q, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663349243 (1h43m16.444282624s) [noise=true , address=104.36.82.54:4166 , piece_id=PDKUUD2O4C6G4NOCZGKI5CDD6GFWCHJISJSYEA7CYI22DIZEGSUQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: rpc: dial tcp 104.36.82.54:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663492963 (1h43m16.444426345s) [noise=true , address=104.36.82.47:4523 , piece_id=CTYLPPTWOPDNEBFNBT62ADKLLOB4A7DUKARJEVDGTLJ3WTQJMCIQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 159.223.143.39:36124->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663529961 (1h43m16.444463339s) [noise=true , address=104.36.82.52:4106 , piece_id=3WNFALTDWUD2VELI3XJKOHZ44OSZZPBHIWWV3DLYHJ77UQCU3XIQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: write tcp 159.223.143.39:39964->104.36.82.52:4106: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663585416 (1h43m16.444518824s) [seg_pos={0 0}, upload=1568]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:35:00.706519445 (1h43m16.487452811s) [upload=1568]: Done uploading segments... Jul 26 23:35:00.706551058 (1h43m16.487484425s) [upload=1568]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.048954187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [...................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.057867129s FAIL (2025-07-26 23:34:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.057894643s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X.X..........XX...X...X..XX.X...X..X.......X...X....XX....] duration: 14.888515279s FAIL (2025-07-26 23:34:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.888544156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.028445732s FAIL (2025-07-26 23:34:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Covw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367198603 (1h41m56.148131979s) [noise=true , address=104.36.82.5:4839 , piece_id=USX2JFKUV34UZUV4VU6KUBGPUK7K6FITP574RNCEXBEWKKVADKJA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:48926->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367235055 (1h41m56.148168436s) [seg_pos={0 0}, upload=1557]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:33:40.418644019 (1h41m56.199577399s) [upload=1557]: Done uploading segments... Jul 26 23:33:40.418702283 (1h41m56.199635662s) [upload=1557]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:34:10.117611634 (1h42m25.898544996s) [upload=1569]: Starting upload Jul 26 23:34:10.117710905 (1h42m25.898644283s) [upload=1569]: Done starting upload Jul 26 23:34:10.117866573 (1h42m25.898799943s) [upload=1569]: Uploading segments... Jul 26 23:34:10.118336221 (1h42m25.899269633s) [upload=1569]: Got next segment. Inline: false Jul 26 23:34:10.118407651 (1h42m25.899341028s) [upload=1569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:34:10.118415198 (1h42m25.899348586s) [upload=1569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:34:10.151847318 (1h42m25.932780734s) [seg_pos={0 0}, upload=1569]: Begin upload segment... Jul 26 23:34:10.151925735 (1h42m25.932859114s) [seg_pos={0 0}, upload=1569]: Done begin upload segment. Jul 26 23:34:10.151963567 (1h42m25.932896959s) [noise=true , address=45.91.116.5:4298 , piece_id=GIAYF6FZRWFG7SUCCBPFDAJFNSH3IIRHL3IRVYOD33VWJ3YOTTWQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.151980790 (1h42m25.932914169s) [noise=true , address=45.91.116.5:4297 , piece_id=NDI2Z6KM7WDCSN2XR52M6LCF53ECEI5MH2D7OUOKADOMVFK2FMRQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.152588724 (1h42m25.933522108s) [noise=true , address=45.91.116.9:4261 , piece_id=YRS5P675XSJUPIDMPBEABQ6TPKA2LKXEHQJ2POAZI2GWTPCUPRYQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.152729338 (1h42m25.933662726s) [noise=true , address=77.90.31.19:4842 , piece_id=XXQA2KLI7OTTNZJVXPCAB4QIJA4FH5KPEN7O77OYQIQIFNODXU5A, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.152939491 (1h42m25.933872874s) [noise=true , address=45.91.116.6:4974 , piece_id=WXG6DG6WQ2S74JYOPBJEXXRCQSTISMMYGKACXIH72IYI5EW6JLXQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153200339 (1h42m25.934133877s) [noise=true , address=45.91.116.12:4227 , piece_id=HY54R26JDN2XVDXXJKFUTJTK2MUFPG73XOIJFOZ76MXSLZJFSMQA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153330981 (1h42m25.934264362s) [noise=true , address=77.90.31.15:4697 , piece_id=2EQV5BWVN6QFAMMSMPUMUWOTV4TF4AI37LRVB255XOEYYCZTTMAA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153670583 (1h42m25.934603973s) [noise=true , address=104.36.82.10:4078 , piece_id=SNMFIG3O3ETCRHCPOZVNSVTRCOVJBJUYI63M7DCFA5R2BNTVQRJQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153691435 (1h42m25.934624799s) [noise=true , address=45.91.116.7:4010 , piece_id=AIM76ZBZCXJZDTHX7MN3V5K3PP4MZGR2ZCO236G2NN75W6XIH53A, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.154154309 (1h42m25.935087696s) [noise=true , address=104.36.82.2:4730 , piece_id=GVJQ5BKZJNDNUS6YFGTSKDTLRASCKRVNE6NTQGTKPB5FIVAJMQPA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.154642256 (1h42m25.935575643s) [noise=true , address=45.91.116.12:4225 , piece_id=QQNIUZYBMGM4DIIVEWNXNNOPRSYOUAD5JUDZ5ZDBGYZD3VKVRATA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.155137383 (1h42m25.936070762s) [noise=true , address=45.91.116.6:4975 , piece_id=46VNMTMFVXMKHY2X52WGM2TMF6SOWHHDVYE2PDH34G5477XJVRDQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.155551019 (1h42m25.936484396s) [noise=true , address=45.91.116.13:4153 , piece_id=V3B6S53NAQI2U7U2S4VGUBOXDQTUCVSD25EMV4RF27A4OVK7OF3A, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.156101107 (1h42m25.937034489s) [noise=true , address=104.36.82.12:4150 , piece_id=X2K72WBWRBLTW4EE2ILEAMPTRQIHMRWXFMWMK2L6YL5HOTBHCZ5Q, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.156611794 (1h42m25.93754518s) [noise=true , address=45.91.116.4:4939 , piece_id=54UNTXSCAEYVF2ZP4GJY44L2UL4SPBQLXEEJ7V6RJH2YYXN7K7BQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.157022833 (1h42m25.93795622s) [noise=true , address=104.36.82.16:4294 , piece_id=Z7AC73U2VQPXOQC75WIMGVVFVYWXGZCIYTYPJZWUOHD2MAN4RBGA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.278956870 (1h42m26.05989024s) [upload=1569]: Next returned nil segment Jul 26 23:34:10.278976555 (1h42m26.059909937s) [upload=1569]: Waiting for error group managing segments... Jul 26 23:34:10.283273193 (1h42m26.064206589s) [noise=true , address=45.91.116.4:4939 , piece_id=54UNTXSCAEYVF2ZP4GJY44L2UL4SPBQLXEEJ7V6RJH2YYXN7K7BQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.283736322 (1h42m26.064669691s) [noise=true , address=45.91.116.6:4974 , piece_id=WXG6DG6WQ2S74JYOPBJEXXRCQSTISMMYGKACXIH72IYI5EW6JLXQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284133013 (1h42m26.065066382s) [noise=true , address=45.91.116.12:4227 , piece_id=HY54R26JDN2XVDXXJKFUTJTK2MUFPG73XOIJFOZ76MXSLZJFSMQA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284529969 (1h42m26.065463332s) [noise=true , address=77.90.31.15:4697 , piece_id=2EQV5BWVN6QFAMMSMPUMUWOTV4TF4AI37LRVB255XOEYYCZTTMAA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284962625 (1h42m26.065896004s) [noise=true , address=45.91.116.13:4153 , piece_id=V3B6S53NAQI2U7U2S4VGUBOXDQTUCVSD25EMV4RF27A4OVK7OF3A, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284990021 (1h42m26.065923393s) [noise=true , address=45.91.116.7:4010 , piece_id=AIM76ZBZCXJZDTHX7MN3V5K3PP4MZGR2ZCO236G2NN75W6XIH53A, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.286004298 (1h42m26.066937667s) [noise=true , address=45.91.116.6:4975 , piece_id=46VNMTMFVXMKHY2X52WGM2TMF6SOWHHDVYE2PDH34G5477XJVRDQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.301316119 (1h42m26.082249509s) [noise=true , address=45.91.116.12:4225 , piece_id=QQNIUZYBMGM4DIIVEWNXNNOPRSYOUAD5JUDZ5ZDBGYZD3VKVRATA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.550182785 (1h42m26.331116155s) [noise=true , address=45.91.116.9:4261 , piece_id=YRS5P675XSJUPIDMPBEABQ6TPKA2LKXEHQJ2POAZI2GWTPCUPRYQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.719621993 (1h42m26.50055536s) [noise=true , address=45.91.116.5:4298 , piece_id=GIAYF6FZRWFG7SUCCBPFDAJFNSH3IIRHL3IRVYOD33VWJ3YOTTWQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.733586459 (1h42m26.51451983s) [noise=true , address=45.91.116.5:4297 , piece_id=NDI2Z6KM7WDCSN2XR52M6LCF53ECEI5MH2D7OUOKADOMVFK2FMRQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:12.318890457 (1h42m28.099823825s) [noise=true , address=77.90.31.19:4842 , piece_id=XXQA2KLI7OTTNZJVXPCAB4QIJA4FH5KPEN7O77OYQIQIFNODXU5A, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:40.057141063 (1h42m55.83807444s) [noise=true , address=104.36.82.10:4078 , piece_id=SNMFIG3O3ETCRHCPOZVNSVTRCOVJBJUYI63M7DCFA5R2BNTVQRJQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057236375 (1h42m55.838169741s) [noise=true , address=104.36.82.2:4730 , piece_id=GVJQ5BKZJNDNUS6YFGTSKDTLRASCKRVNE6NTQGTKPB5FIVAJMQPA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057327362 (1h42m55.838260728s) [noise=true , address=104.36.82.16:4294 , piece_id=Z7AC73U2VQPXOQC75WIMGVVFVYWXGZCIYTYPJZWUOHD2MAN4RBGA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057396771 (1h42m55.838330137s) [noise=true , address=104.36.82.12:4150 , piece_id=X2K72WBWRBLTW4EE2ILEAMPTRQIHMRWXFMWMK2L6YL5HOTBHCZ5Q, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:40600->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057416370 (1h42m55.838349747s) [seg_pos={0 0}, upload=1569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:34:40.082099388 (1h42m55.863032755s) [upload=1569]: Done uploading segments... Jul 26 23:34:40.082136123 (1h42m55.863069512s) [upload=1569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028571482s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X.X..........XX...X...X..XX.X...X..X.......X...X....XX...] duration: 43.630721457s FAIL (2025-07-26 23:34:28 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.63074948s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........................XXX.X....XX.X.X.X.......XXX...XX....] duration: 2.014707975s FAIL (2025-07-26 23:34:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.014764827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031070205s FAIL (2025-07-26 23:34:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== WQFDADOX3EAQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.641611597 (1h41m16.42254498s) [noise=true , address=77.90.31.19:4871 , piece_id=JBUXIBPSEADTTRVUJ3E4HN7UCFGE5Y4SZ4CTU5AVZW7AABTM6IFQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.641967837 (1h41m16.422901231s) [noise=true , address=104.36.82.47:4530 , piece_id=WKQRVPUDU3GWIRDHHRSELERDHCLX6FGDUGTEIIEAO33LSZSJOCOQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.642685688 (1h41m16.423619082s) [noise=true , address=77.90.31.18:4821 , piece_id=NOR4OYLYORUBVORQX6T7HQEEIWFANUP7FZM736VPMOIQWU7IRDYQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.642890793 (1h41m16.423824177s) [noise=true , address=104.36.82.54:4177 , piece_id=62HU5KO42VQL7B25ZLIEOXW5FV5ALO6Y5UTXCSJVXS6MTFSDTIMQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643487814 (1h41m16.424421194s) [noise=true , address=79.127.201.215:4355 , piece_id=SXPU2BGAM7MYIWR45FPIHI4WCKHTQKBEKDOLALUPMLJGHZ6IU23Q, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643536526 (1h41m16.424469912s) [noise=true , address=77.90.31.17:4787 , piece_id=EP7BFHHIFK2V7FAEK2EIN5SZLCHDDVAKT37U2AHUHHIXOTVNEY4A, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643940021 (1h41m16.424873411s) [noise=true , address=79.127.205.234:4607 , piece_id=PZ7OYTUK5IHIYTVDSTXQYRZEWOKHG3SNUJNY75P4G3VCGPFMI6BQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643966746 (1h41m16.424900129s) [noise=true , address=79.127.220.98:4422 , piece_id=WAVGYZ43ZKCDLXP7OVTRH6SIFPMOGGQE2DB4KJEGU3F7IBDBWBOQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.644465625 (1h41m16.425399087s) [noise=true , address=104.36.82.46:4513 , piece_id=X6JAJ7LP52L2ZU6H6POXT2SIEOKV4R27FUQTJOTO24TO2CJMKHNA, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.644498294 (1h41m16.425431679s) [noise=true , address=104.36.82.22:4514 , piece_id=2YDELYDN44OES2KUVXHVBQ3ZRLOJ4G7PAKGRBVBDY4UXMFRHJPLQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.644942162 (1h41m16.425875551s) [noise=true , address=64.49.13.137:4502 , piece_id=GDV67RIJCJV6XKGOPTHLZLXZRLYRB7X57WM7LYM6AKXDC4QB4F6A, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.645340487 (1h41m16.426273876s) [noise=true , address=77.90.31.12:4602 , piece_id=UETVWAEM64VE6ELYYKCDNKPIBAXRJNOTK2AS6DZI26P53FL3YXSA, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.645726110 (1h41m16.4266595s) [noise=true , address=104.36.82.55:4202 , piece_id=HCPA3WWF7POYGJCYZC5Z7QBRELSOG7G4LVPD6PSJX4BNTRCK7B3Q, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.647326218 (1h41m16.428259607s) [noise=true , address=79.127.219.38:4005 , piece_id=2KIIIHAK2F4FSIHHFKNSJ2OX4WG777PKRE3YCNXMBDHHPHNOE7AA, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.647907593 (1h41m16.428840955s) [noise=true , address=77.90.31.21:4906 , piece_id=YISCG44QDDGPOBUGQTMBXT2VFYADKCZJ2N2JRUZGZVNBIJLUJ3VQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.750777473 (1h41m16.531710852s) [upload=1554]: Next returned nil segment Jul 26 23:33:00.750797755 (1h41m16.531731123s) [upload=1554]: Waiting for error group managing segments... Jul 26 23:33:00.762018232 (1h41m16.542951597s) [noise=true , address=79.127.219.38:4005 , piece_id=2KIIIHAK2F4FSIHHFKNSJ2OX4WG777PKRE3YCNXMBDHHPHNOE7AA, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:00.775914020 (1h41m16.556847399s) [noise=true , address=77.90.31.21:4906 , piece_id=YISCG44QDDGPOBUGQTMBXT2VFYADKCZJ2N2JRUZGZVNBIJLUJ3VQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:00.837314740 (1h41m16.618248105s) [noise=true , address=156.146.43.225:4434 , piece_id=TDWDSLZMIKDYDDH43NQNL3N4QZCUOSOETEGIQV766XFPBLFG74PQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:33:01.233662901 (1h41m17.014596278s) [noise=true , address=79.127.205.234:4607 , piece_id=PZ7OYTUK5IHIYTVDSTXQYRZEWOKHG3SNUJNY75P4G3VCGPFMI6BQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:01.986856055 (1h41m17.767789441s) [noise=true , address=64.49.13.137:4502 , piece_id=GDV67RIJCJV6XKGOPTHLZLXZRLYRB7X57WM7LYM6AKXDC4QB4F6A, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 26 23:33:03.615690829 (1h41m19.396624196s) [noise=true , address=79.127.220.98:4422 , piece_id=WAVGYZ43ZKCDLXP7OVTRH6SIFPMOGGQE2DB4KJEGU3F7IBDBWBOQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:07.962746934 (1h41m23.743680302s) [noise=true , address=77.90.31.19:4871 , piece_id=JBUXIBPSEADTTRVUJ3E4HN7UCFGE5Y4SZ4CTU5AVZW7AABTM6IFQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:13.197228992 (1h41m28.978162361s) [noise=true , address=79.127.201.215:4355 , piece_id=SXPU2BGAM7MYIWR45FPIHI4WCKHTQKBEKDOLALUPMLJGHZ6IU23Q, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:34:00.567666622 (1h42m16.348600005s) [noise=true , address=104.36.82.54:4177 , piece_id=62HU5KO42VQL7B25ZLIEOXW5FV5ALO6Y5UTXCSJVXS6MTFSDTIMQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: rpc: dial tcp 104.36.82.54:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.567774109 (1h42m16.348707477s) [noise=true , address=104.36.82.55:4202 , piece_id=HCPA3WWF7POYGJCYZC5Z7QBRELSOG7G4LVPD6PSJX4BNTRCK7B3Q, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.567858950 (1h42m16.348792332s) [noise=true , address=104.36.82.47:4530 , piece_id=WKQRVPUDU3GWIRDHHRSELERDHCLX6FGDUGTEIIEAO33LSZSJOCOQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.567927527 (1h42m16.348860906s) [noise=true , address=77.90.31.17:4787 , piece_id=EP7BFHHIFK2V7FAEK2EIN5SZLCHDDVAKT37U2AHUHHIXOTVNEY4A, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568002098 (1h42m16.348935461s) [noise=true , address=104.36.82.22:4514 , piece_id=2YDELYDN44OES2KUVXHVBQ3ZRLOJ4G7PAKGRBVBDY4UXMFRHJPLQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568160727 (1h42m16.34909409s) [noise=true , address=77.90.31.12:4602 , piece_id=UETVWAEM64VE6ELYYKCDNKPIBAXRJNOTK2AS6DZI26P53FL3YXSA, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568233155 (1h42m16.349166523s) [noise=true , address=104.36.82.53:4152 , piece_id=GYMX5GSFR7SATHB7PLZM2PTR5F5RUOJSS3Y4YREWWQFDADOX3EAQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 159.223.143.39:60046->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568407586 (1h42m16.34934095s) [noise=true , address=104.36.82.46:4513 , piece_id=X6JAJ7LP52L2ZU6H6POXT2SIEOKV4R27FUQTJOTO24TO2CJMKHNA, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:34:00.568556099 (1h42m16.349489463s) [noise=true , address=77.90.31.18:4821 , piece_id=NOR4OYLYORUBVORQX6T7HQEEIWFANUP7FZM736VPMOIQWU7IRDYQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:34:00.568752939 (1h42m16.34968632s) [noise=true , address=104.36.82.56:4246 , piece_id=RLKKQRD4Q4TQZ5DJCTWEB4KOD2IZYB6ONKACYIWN76VD5JVI533Q, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1XtKegw9zcPDGZu9z7STKgAzwx5KFrkB9qCgdNUe6XqZ6tzzj9, address:104.36.82.56:4246): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:34:00.568775348 (1h42m16.34970873s) [seg_pos={0 0}, upload=1554]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:34:00.595410818 (1h42m16.376344187s) [upload=1554]: Done uploading segments... Jul 26 23:34:00.595443660 (1h42m16.376377032s) [upload=1554]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.031090347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [....................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026668982s FAIL (2025-07-26 23:33:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.026721375s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [....X.X..........XX...X...X..XX.X...X..X.......X...X....XX..] duration: 45.511306563s FAIL (2025-07-26 23:33:44 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.511338436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........................XXX.X....XX.X.X.X.......XXX...XX...] duration: 637.747882ms FAIL (2025-07-26 23:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 637.778114ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.054525039s FAIL (2025-07-26 23:33:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== ]: Done starting upload Jul 26 23:33:10.399859122 (1h41m26.180792491s) [upload=1557]: Uploading segments... Jul 26 23:33:10.399921012 (1h41m26.1808544s) [upload=1557]: Got next segment. Inline: false Jul 26 23:33:10.399963477 (1h41m26.180896863s) [upload=1557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:33:10.399970489 (1h41m26.180903875s) [upload=1557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:33:10.427358387 (1h41m26.208291775s) [seg_pos={0 0}, upload=1557]: Begin upload segment... Jul 26 23:33:10.427454451 (1h41m26.208387813s) [seg_pos={0 0}, upload=1557]: Done begin upload segment. Jul 26 23:33:10.427500329 (1h41m26.208433713s) [noise=true , address=45.91.116.6:4975 , piece_id=DMZSXLD7QRBSM5O46R23HMOXNTHVJN7I3RGR44GYRUIZ3SHHDUZA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.427521509 (1h41m26.208454894s) [noise=true , address=104.36.82.16:4293 , piece_id=4O5IUHFBXARTXNMYUQT4YAS2II2R7YHCCZ5N6VDZJQ2IMOWGVAKQ, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428258294 (1h41m26.209191683s) [noise=true , address=104.36.82.16:4295 , piece_id=ORU44S4XE4UPECW6ZHMTDVAQEWKV7WCK4YUALWR2NWCKVP6OXAPA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428278023 (1h41m26.209211384s) [noise=true , address=104.36.82.9:4005 , piece_id=4GY4REWU4USJ7CRTV4GVDXXLB52ELWT67FTY4CLBZTMQJSVBE4TQ, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428671971 (1h41m26.209605349s) [noise=true , address=45.91.116.9:4261 , piece_id=R47BU7WN26UUP6PA6DKFOS2RVY2HZX6W5OG2AMR22RJEQXQIQTPA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428753539 (1h41m26.209686908s) [noise=true , address=45.91.116.13:4153 , piece_id=B7HORTLQWCV7QJBNIUT3G6CNNFPCX3DW3LEUGESGJRWAVW77X2OA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.429075533 (1h41m26.210008914s) [noise=true , address=45.91.116.5:4297 , piece_id=UZ5ARATYV422WL2QODOIJ2HCOZ7Q4BVWCY7VIW2KJBA3A443CPZA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.429561053 (1h41m26.210494431s) [noise=true , address=104.36.82.13:4187 , piece_id=J2FC4MOJOEAWU6GRSBOTGJ7JPXRZLXF37YEJVWSYESWPST3VXWNQ, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430018688 (1h41m26.210952066s) [noise=true , address=104.36.82.5:4839 , piece_id=USX2JFKUV34UZUV4VU6KUBGPUK7K6FITP574RNCEXBEWKKVADKJA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430069970 (1h41m26.21100334s) [noise=true , address=104.36.82.12:4150 , piece_id=J3MGMI5FC26BHG2GQE2YAVZEH3ZULCF2IYNWX6CWBQGJ3WPPDOXA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430442592 (1h41m26.21137598s) [noise=true , address=104.36.82.8:4969 , piece_id=G5TLXWEUIKPG2V7PTVYWPMRVTULCW3O7I74VUKIMVWNSIXMBUP6Q, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430815607 (1h41m26.211749001s) [noise=true , address=104.36.82.7:4912 , piece_id=NNJN2XBUJ2SZLGKPN76RJCWZVFGFJ4KDR3BVO3QZXZ3S3KJDPWGA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.431084578 (1h41m26.212017962s) [noise=true , address=45.91.116.14:4189 , piece_id=FQHSP72JSQMQZ5UVPVIJ4QNBCWFLD2KQJYNXPLKCCQRFBEBG2PBA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.431634704 (1h41m26.212568086s) [noise=true , address=45.91.116.14:4191 , piece_id=WVPCSQLMPY7F4DILLY56OFUFALZEX34R3RLIDJHMER26FLP4J5NA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.432565394 (1h41m26.21349877s) [noise=true , address=45.91.116.12:4227 , piece_id=FG2ZVMM72EZJH4SHSIYGYT5TPCQNYJNDBKAMGVFMII5AW7DZV2IA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.433293254 (1h41m26.21422662s) [noise=true , address=104.36.82.12:4149 , piece_id=HFGQU3APX4IO6IY2FP43OIXA5UFO4J3O5TKUAEFESV3465VVEBUA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.532771490 (1h41m26.313704867s) [upload=1557]: Next returned nil segment Jul 26 23:33:10.532791528 (1h41m26.313724915s) [upload=1557]: Waiting for error group managing segments... Jul 26 23:33:10.537134935 (1h41m26.318068298s) [noise=true , address=45.91.116.12:4227 , piece_id=FG2ZVMM72EZJH4SHSIYGYT5TPCQNYJNDBKAMGVFMII5AW7DZV2IA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.537246745 (1h41m26.318180114s) [noise=true , address=45.91.116.6:4975 , piece_id=DMZSXLD7QRBSM5O46R23HMOXNTHVJN7I3RGR44GYRUIZ3SHHDUZA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.537693491 (1h41m26.318626855s) [noise=true , address=45.91.116.13:4153 , piece_id=B7HORTLQWCV7QJBNIUT3G6CNNFPCX3DW3LEUGESGJRWAVW77X2OA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.673740835 (1h41m26.454674213s) [noise=true , address=45.91.116.5:4297 , piece_id=UZ5ARATYV422WL2QODOIJ2HCOZ7Q4BVWCY7VIW2KJBA3A443CPZA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.687782734 (1h41m26.468716115s) [noise=true , address=45.91.116.9:4261 , piece_id=R47BU7WN26UUP6PA6DKFOS2RVY2HZX6W5OG2AMR22RJEQXQIQTPA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.773600304 (1h41m26.55453367s) [noise=true , address=45.91.116.14:4191 , piece_id=WVPCSQLMPY7F4DILLY56OFUFALZEX34R3RLIDJHMER26FLP4J5NA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.780943150 (1h41m26.561876529s) [noise=true , address=45.91.116.14:4189 , piece_id=FQHSP72JSQMQZ5UVPVIJ4QNBCWFLD2KQJYNXPLKCCQRFBEBG2PBA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:40.366858867 (1h41m56.147792246s) [noise=true , address=104.36.82.16:4295 , piece_id=ORU44S4XE4UPECW6ZHMTDVAQEWKV7WCK4YUALWR2NWCKVP6OXAPA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.366938462 (1h41m56.147871826s) [noise=true , address=104.36.82.9:4005 , piece_id=4GY4REWU4USJ7CRTV4GVDXXLB52ELWT67FTY4CLBZTMQJSVBE4TQ, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367052012 (1h41m56.147985388s) [noise=true , address=104.36.82.12:4149 , piece_id=HFGQU3APX4IO6IY2FP43OIXA5UFO4J3O5TKUAEFESV3465VVEBUA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367068681 (1h41m56.148002047s) [noise=true , address=104.36.82.13:4187 , piece_id=J2FC4MOJOEAWU6GRSBOTGJ7JPXRZLXF37YEJVWSYESWPST3VXWNQ, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367124295 (1h41m56.148057673s) [noise=true , address=104.36.82.16:4293 , piece_id=4O5IUHFBXARTXNMYUQT4YAS2II2R7YHCCZ5N6VDZJQ2IMOWGVAKQ, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367163255 (1h41m56.148096621s) [noise=true , address=104.36.82.12:4150 , piece_id=J3MGMI5FC26BHG2GQE2YAVZEH3ZULCF2IYNWX6CWBQGJ3WPPDOXA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:50168->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367174765 (1h41m56.148108133s) [noise=true , address=104.36.82.8:4969 , piece_id=G5TLXWEUIKPG2V7PTVYWPMRVTULCW3O7I74VUKIMVWNSIXMBUP6Q, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367188533 (1h41m56.148121906s) [noise=true , address=104.36.82.7:4912 , piece_id=NNJN2XBUJ2SZLGKPN76RJCWZVFGFJ4KDR3BVO3QZXZ3S3KJDPWGA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367198603 (1h41m56.148131979s) [noise=true , address=104.36.82.5:4839 , piece_id=USX2JFKUV34UZUV4VU6KUBGPUK7K6FITP574RNCEXBEWKKVADKJA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:48926->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367235055 (1h41m56.148168436s) [seg_pos={0 0}, upload=1557]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:33:40.418644019 (1h41m56.199577399s) [upload=1557]: Done uploading segments... Jul 26 23:33:40.418702283 (1h41m56.199635662s) [upload=1557]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.054554625s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................XXX.X....XX.X.X.X.......XXX...XX..] duration: 1.933483184s FAIL (2025-07-26 23:33:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.934159436s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [.XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030649888s FAIL (2025-07-26 23:33:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== upload=1537]: Uploading piece... Jul 26 23:32:00.503791147 (1h40m16.284724536s) [noise=true , address=156.146.43.225:4448 , piece_id=WCQMBCF2CUOSVQASPGY6THK2PACIDKPLLHAEHIINOOI66IOCY76Q, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.504200430 (1h40m16.285133819s) [noise=true , address=77.90.31.10:4523 , piece_id=RTT3CJBI7WVM4PMMU5RXVOFXVF5XST7MA73SJZGH2QXUA7FUFJCQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.504606966 (1h40m16.28554035s) [noise=true , address=79.127.205.234:4605 , piece_id=MJRU5HAPBF6ORRB4Z6FFWSI4KNMPIOOGRQMYGRSQOOUBYYP3AKUQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.504990011 (1h40m16.285923394s) [noise=true , address=77.90.31.18:4831 , piece_id=7AGKK7XRXGSRCDVU3YJGS64OVZ6NRQWUSOWGKGKSA4F3DPKUUKUQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.505360782 (1h40m16.286294163s) [noise=true , address=79.127.201.215:4344 , piece_id=XIKKXCYS46JFDZW7RMNC2HIVNC3YWB6THQ2TJAMNVJ4ULFCHTCKQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.505733748 (1h40m16.286667112s) [noise=true , address=104.36.82.46:4496 , piece_id=373IDZNHVH6QHAY6KZDDEFO2IDB4FFHCNFF7KXHDF6AOV366PHUQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.506097792 (1h40m16.287031192s) [noise=true , address=104.36.82.54:4167 , piece_id=PYTSSV6KF2RDQGCSAA26LFU5CM7FERKLMIHNBNAUUXJYV54F47GA, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.506503500 (1h40m16.287436889s) [noise=true , address=104.36.82.10:4084 , piece_id=YJBAKYBYGA7ZNSKTKZHSPLXDBULTRQNXZ6P5FVQIBZ6E6QUDDIQQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.506884663 (1h40m16.287818053s) [noise=true , address=104.36.82.47:4532 , piece_id=G7V3LGUEFMNTJ5KRDHH6J5Y2VSAPJ6GFSR7Q7VNOBK5GXWJXGU7Q, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.507249436 (1h40m16.28818282s) [noise=true , address=45.91.116.9:4277 , piece_id=OX6EYFNEJCSQD7RDYEF7GDHKAAUNHN237EKYHPXQ465UW4YISCOA, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.507562406 (1h40m16.288495794s) [noise=true , address=77.90.31.21:4893 , piece_id=NL3ZRIJKSMILTMNUW2RGQLHE6OOH2EARLIBAMGQUKUWNWTAVPATQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.507994281 (1h40m16.288927672s) [noise=true , address=104.36.82.53:4147 , piece_id=MHQE4C2UOOKWTOSIC3VT4DOG74BWAI225JW3SF7WSXLHZ6OR4H6A, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.508539751 (1h40m16.28947314s) [noise=true , address=77.90.31.15:4723 , piece_id=B3OPFMMS3DBNKXGGPZAXYI7VKOSDEMJFO3I2QLLIF4CNYGGM3XBQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.509112523 (1h40m16.290045906s) [noise=true , address=104.36.82.55:4199 , piece_id=UIUKCUJY3RIV3KNTAOYZPPOUCNKJKND3BBQSNHW2D2IN5BANTZ6A, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.509543680 (1h40m16.29047706s) [noise=true , address=64.49.13.140:4508 , piece_id=EIHQTWDL72BNUCHSLGILXSZRREXXXBBRIUC3L5S3SZ6I37UYSFAQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.510018545 (1h40m16.290951924s) [noise=true , address=104.36.82.56:4230 , piece_id=5LBVPKYLT652M2K4FIDNJGIHU3R7Z4FW3UPGITE5RZCNQMOHGZJQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.677960694 (1h40m16.458894073s) [upload=1537]: Next returned nil segment Jul 26 23:32:00.677980641 (1h40m16.458914021s) [upload=1537]: Waiting for error group managing segments... Jul 26 23:32:00.724065189 (1h40m16.504998561s) [noise=true , address=77.90.31.10:4523 , piece_id=RTT3CJBI7WVM4PMMU5RXVOFXVF5XST7MA73SJZGH2QXUA7FUFJCQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:00.724663736 (1h40m16.505597113s) [noise=true , address=77.90.31.21:4893 , piece_id=NL3ZRIJKSMILTMNUW2RGQLHE6OOH2EARLIBAMGQUKUWNWTAVPATQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:00.724957173 (1h40m16.505890538s) [noise=true , address=77.90.31.15:4723 , piece_id=B3OPFMMS3DBNKXGGPZAXYI7VKOSDEMJFO3I2QLLIF4CNYGGM3XBQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:01.121129125 (1h40m16.902062519s) [noise=true , address=79.127.205.234:4605 , piece_id=MJRU5HAPBF6ORRB4Z6FFWSI4KNMPIOOGRQMYGRSQOOUBYYP3AKUQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:01.397274599 (1h40m17.178207973s) [noise=true , address=79.127.219.47:4048 , piece_id=RB3ABLIP67YOFAHF2CDS2STPDURXAJZG2FP5GO2K2FBBS4B6DKNQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:02.379368187 (1h40m18.160301558s) [noise=true , address=45.91.116.9:4277 , piece_id=OX6EYFNEJCSQD7RDYEF7GDHKAAUNHN237EKYHPXQ465UW4YISCOA, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:02.738944034 (1h40m18.519877404s) [noise=true , address=156.146.43.225:4448 , piece_id=WCQMBCF2CUOSVQASPGY6THK2PACIDKPLLHAEHIINOOI66IOCY76Q, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:04.607773367 (1h40m20.388706752s) [noise=true , address=64.49.13.140:4508 , piece_id=EIHQTWDL72BNUCHSLGILXSZRREXXXBBRIUC3L5S3SZ6I37UYSFAQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:32:33.205246417 (1h40m48.986179795s) [noise=true , address=79.127.201.215:4344 , piece_id=XIKKXCYS46JFDZW7RMNC2HIVNC3YWB6THQ2TJAMNVJ4ULFCHTCKQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:33:00.428477210 (1h41m16.20941059s) [noise=true , address=104.36.82.46:4496 , piece_id=373IDZNHVH6QHAY6KZDDEFO2IDB4FFHCNFF7KXHDF6AOV366PHUQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:55896->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.428848850 (1h41m16.209782227s) [noise=true , address=104.36.82.56:4230 , piece_id=5LBVPKYLT652M2K4FIDNJGIHU3R7Z4FW3UPGITE5RZCNQMOHGZJQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.428934177 (1h41m16.209867545s) [noise=true , address=104.36.82.54:4167 , piece_id=PYTSSV6KF2RDQGCSAA26LFU5CM7FERKLMIHNBNAUUXJYV54F47GA, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: rpc: dial tcp 104.36.82.54:4167: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429006265 (1h41m16.209939628s) [noise=true , address=104.36.82.55:4199 , piece_id=UIUKCUJY3RIV3KNTAOYZPPOUCNKJKND3BBQSNHW2D2IN5BANTZ6A, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429159614 (1h41m16.210092991s) [noise=true , address=77.90.31.18:4831 , piece_id=7AGKK7XRXGSRCDVU3YJGS64OVZ6NRQWUSOWGKGKSA4F3DPKUUKUQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: rpc: dial tcp 77.90.31.18:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429222931 (1h41m16.210156312s) [noise=true , address=104.36.82.52:4118 , piece_id=ANUDW6CVC52AZ6RA4ULRA46UIMC7L5PV4YKOTBSRQMMZAYSOOT4Q, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: rpc: dial tcp 104.36.82.52:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429250400 (1h41m16.210183782s) [noise=true , address=104.36.82.47:4532 , piece_id=G7V3LGUEFMNTJ5KRDHH6J5Y2VSAPJ6GFSR7Q7VNOBK5GXWJXGU7Q, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429284001 (1h41m16.210217369s) [noise=true , address=104.36.82.10:4084 , piece_id=YJBAKYBYGA7ZNSKTKZHSPLXDBULTRQNXZ6P5FVQIBZ6E6QUDDIQQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1mELnhMiRJ5GkUWYEggdxhCQ2JnDbT7qEMSiAuMQsF7BTnwP1u, address:104.36.82.10:4084): protocol: rpc: dial tcp 104.36.82.10:4084: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429332105 (1h41m16.210265483s) [noise=true , address=104.36.82.53:4147 , piece_id=MHQE4C2UOOKWTOSIC3VT4DOG74BWAI225JW3SF7WSXLHZ6OR4H6A, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 159.223.143.39:59358->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429377719 (1h41m16.210311112s) [seg_pos={0 0}, upload=1537]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:33:00.457182466 (1h41m16.238115845s) [upload=1537]: Done uploading segments... Jul 26 23:33:00.457215184 (1h41m16.238148563s) [upload=1537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.03068491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032731352s FAIL (2025-07-26 23:32:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== .353122105 (1h40m19.134055493s) [seg_pos={0 0}, upload=1538]: Begin upload segment... Jul 26 23:32:03.353239411 (1h40m19.134172801s) [seg_pos={0 0}, upload=1538]: Done begin upload segment. Jul 26 23:32:03.353260643 (1h40m19.13419404s) [noise=true , address=104.36.82.10:4078 , piece_id=T4XORQPGNCLYENW54QLBPDUAHQ7S2X6HFKGFCEJC6QDSZBV772YA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.353274424 (1h40m19.134207807s) [noise=true , address=45.91.116.4:4939 , piece_id=3KRX5PB2OSEIGFNO3UFIWLXTQJZU3KARWVYZRZK7BBI24KNWV2UA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.353802423 (1h40m19.13473581s) [noise=true , address=45.91.116.6:4974 , piece_id=EZTTBOCNKNHFJEZVZ2ZXPIQEXX4533Q5APAUBJAJOUBJG7QY5N4A, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.353935259 (1h40m19.134868648s) [noise=true , address=45.91.116.5:4298 , piece_id=H27L56HRWWIQ4YAWTCTM5SDYAY5UNAWAZZ75RJG6KGHESMX7Y7ZQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.355345572 (1h40m19.136278956s) [noise=true , address=104.36.82.13:4187 , piece_id=XOXI42ULDEDSO3JTPCHVJHVIGMGURJ5L2WVZAAX2OUH3JOZPSQLA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.355891971 (1h40m19.136825355s) [noise=true , address=45.91.116.6:4975 , piece_id=HUPUTYUEXVBVWVTOLCM2NUMWJDKYW7WYENA422YB2VZ3SJURVE7Q, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.356479709 (1h40m19.137413098s) [noise=true , address=104.36.82.2:4731 , piece_id=FO7NRU4GEDROJWQIZBGZAEQDKGUH6SIDGC3HYB6AK2F4TNZOUFOQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.357132302 (1h40m19.138065697s) [noise=true , address=104.36.82.8:4970 , piece_id=KDM65323RENUMUPSUXBHPGFE42X73HWDLF4E4IS64OJSHICOAKFQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.358048179 (1h40m19.138981566s) [noise=true , address=104.36.82.11:4115 , piece_id=3Z5T74QYNQUEAFUA3OXUOZPCZUVAXNBNHK4BW4Y3WLQEJQTKVYQA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.360615159 (1h40m19.141548539s) [noise=true , address=104.36.82.14:4221 , piece_id=AYL23B7HRDJYXHDGKC5HSNRQRRX5WFOT7NJBRZIM4UXWG6P5PPGQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.361186918 (1h40m19.142120288s) [noise=true , address=104.36.82.12:4151 , piece_id=JMIOIHXIEI2FEI7S2YXLYH46S3WAOCGXRXSWP3SJF2ONODRUNYYA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.361626661 (1h40m19.14256003s) [noise=true , address=104.36.82.7:4911 , piece_id=KQ7UHVHPW64JOA756ZGELUHK6OSUYLK6DRP4RDFPBIIZWDG2IY5A, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.362033254 (1h40m19.142966617s) [noise=true , address=104.36.82.9:4007 , piece_id=YJOAFWG4SZHNDTB6YC2XD4R3OFX4W6ZACUYEK637AG27IGDXXAXQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.362442429 (1h40m19.143375797s) [noise=true , address=104.36.82.15:4259 , piece_id=BQ7CKKHPF2K3I3IFZYQHOLXDZCVIOTQZJVTXTG4I6OHGSPFXGJIQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.362783491 (1h40m19.143716857s) [noise=true , address=104.36.82.6:4875 , piece_id=IDAQ4QP2KOYZMMFVDETP643A7AWXGXZNAQLW4QFC7YRWNMPJMXRQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.363421811 (1h40m19.144355189s) [noise=true , address=45.91.116.12:4225 , piece_id=LQ3QL7HRFD5V6DCAI4EHQUQLIUWTEURUSPNXIN5CULSQNV4HYKRA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.474840635 (1h40m19.255774015s) [upload=1538]: Next returned nil segment Jul 26 23:32:03.474863766 (1h40m19.255797155s) [upload=1538]: Waiting for error group managing segments... Jul 26 23:32:03.481844144 (1h40m19.262777526s) [noise=true , address=45.91.116.4:4939 , piece_id=3KRX5PB2OSEIGFNO3UFIWLXTQJZU3KARWVYZRZK7BBI24KNWV2UA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.483000440 (1h40m19.26393382s) [noise=true , address=45.91.116.6:4974 , piece_id=EZTTBOCNKNHFJEZVZ2ZXPIQEXX4533Q5APAUBJAJOUBJG7QY5N4A, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.483951106 (1h40m19.264884478s) [noise=true , address=45.91.116.6:4975 , piece_id=HUPUTYUEXVBVWVTOLCM2NUMWJDKYW7WYENA422YB2VZ3SJURVE7Q, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.485508606 (1h40m19.266441987s) [noise=true , address=45.91.116.12:4225 , piece_id=LQ3QL7HRFD5V6DCAI4EHQUQLIUWTEURUSPNXIN5CULSQNV4HYKRA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.782069594 (1h40m19.563002966s) [noise=true , address=45.91.116.5:4298 , piece_id=H27L56HRWWIQ4YAWTCTM5SDYAY5UNAWAZZ75RJG6KGHESMX7Y7ZQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:33.266246032 (1h40m49.047179409s) [noise=true , address=104.36.82.11:4115 , piece_id=3Z5T74QYNQUEAFUA3OXUOZPCZUVAXNBNHK4BW4Y3WLQEJQTKVYQA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266289037 (1h40m49.047222415s) [noise=true , address=104.36.82.8:4970 , piece_id=KDM65323RENUMUPSUXBHPGFE42X73HWDLF4E4IS64OJSHICOAKFQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266337416 (1h40m49.047270796s) [noise=true , address=104.36.82.14:4221 , piece_id=AYL23B7HRDJYXHDGKC5HSNRQRRX5WFOT7NJBRZIM4UXWG6P5PPGQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266394330 (1h40m49.047327691s) [noise=true , address=104.36.82.6:4875 , piece_id=IDAQ4QP2KOYZMMFVDETP643A7AWXGXZNAQLW4QFC7YRWNMPJMXRQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266452241 (1h40m49.047385618s) [noise=true , address=104.36.82.13:4187 , piece_id=XOXI42ULDEDSO3JTPCHVJHVIGMGURJ5L2WVZAAX2OUH3JOZPSQLA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266476946 (1h40m49.04741031s) [noise=true , address=104.36.82.10:4078 , piece_id=T4XORQPGNCLYENW54QLBPDUAHQ7S2X6HFKGFCEJC6QDSZBV772YA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266519005 (1h40m49.047452381s) [noise=true , address=104.36.82.2:4731 , piece_id=FO7NRU4GEDROJWQIZBGZAEQDKGUH6SIDGC3HYB6AK2F4TNZOUFOQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266546742 (1h40m49.047480118s) [noise=true , address=104.36.82.12:4151 , piece_id=JMIOIHXIEI2FEI7S2YXLYH46S3WAOCGXRXSWP3SJF2ONODRUNYYA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266605231 (1h40m49.047538597s) [noise=true , address=104.36.82.7:4911 , piece_id=KQ7UHVHPW64JOA756ZGELUHK6OSUYLK6DRP4RDFPBIIZWDG2IY5A, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:53120->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266616554 (1h40m49.047549915s) [noise=true , address=104.36.82.15:4259 , piece_id=BQ7CKKHPF2K3I3IFZYQHOLXDZCVIOTQZJVTXTG4I6OHGSPFXGJIQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:39390->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266627022 (1h40m49.047560384s) [noise=true , address=104.36.82.9:4007 , piece_id=YJOAFWG4SZHNDTB6YC2XD4R3OFX4W6ZACUYEK637AG27IGDXXAXQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266648756 (1h40m49.047582138s) [seg_pos={0 0}, upload=1538]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:32:33.296633567 (1h40m49.077566933s) [upload=1538]: Done uploading segments... Jul 26 23:32:33.296666311 (1h40m49.077599697s) [upload=1538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.032764262s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........................XXX.X....XX.X.X.X.......XXX...XX.] duration: 662.936088ms FAIL (2025-07-26 23:32:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 662.991151ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.....X.X..........XX...X...X..XX.X...X..X.......X...X....XX.] duration: 13.962058784s FAIL (2025-07-26 23:32:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.962087166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [.....................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 42.537929ms FAIL (2025-07-26 23:32:17 UTC) orbiter warning: uplink: too many requests Duration: 42.563919ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [............................XXX.X....XX.X.X.X.......XXX...XX] duration: 30.001015671s FAIL (2025-07-26 23:32:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001153328s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [......................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.047768683s FAIL (2025-07-26 23:32:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.047800604s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [......X.X..........XX...X...X..XX.X...X..X.......X...X....XX] duration: 1m0.00256466s FAIL (2025-07-26 23:32:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002588778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.04311148s FAIL (2025-07-26 23:32:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== cessful: 12 optimal: 13 errs: <nil> Jul 26 23:30:54.744877179 (1h39m10.525810546s) [upload=1514]: Done uploading segments... Jul 26 23:30:54.744907487 (1h39m10.525840853s) [upload=1514]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:31:31.576458494 (1h39m47.357391884s) [upload=1529]: Starting upload Jul 26 23:31:31.576597271 (1h39m47.357530637s) [upload=1529]: Done starting upload Jul 26 23:31:31.576744624 (1h39m47.357677993s) [upload=1529]: Uploading segments... Jul 26 23:31:31.577178443 (1h39m47.358111832s) [upload=1529]: Got next segment. Inline: false Jul 26 23:31:31.577252019 (1h39m47.358185384s) [upload=1529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:31:31.577260482 (1h39m47.358193867s) [upload=1529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:31:31.608967209 (1h39m47.38990059s) [seg_pos={0 0}, upload=1529]: Begin upload segment... Jul 26 23:31:31.609041313 (1h39m47.389974675s) [seg_pos={0 0}, upload=1529]: Done begin upload segment. Jul 26 23:31:31.609054081 (1h39m47.389987471s) [noise=true , address=77.90.31.15:4698 , piece_id=7ZYN4RXVVEI6BS2JW4IVER7QNYF6JV6SZY3JREKKZUYBNVHQIR6Q, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.609065958 (1h39m47.389999343s) [noise=true , address=77.90.31.18:4805 , piece_id=3Q4F7FHNVURJIXDHTXU55ACLQENASWHGE4RSY744USTVD5DRXBUA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.609564670 (1h39m47.390498058s) [noise=true , address=104.36.82.12:4149 , piece_id=PTPZSM423OKVFTTZBXYZBKYLEU72AQY55QBL2SE5CRRH7QKYHFDA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.609594335 (1h39m47.390527718s) [noise=true , address=45.91.116.9:4263 , piece_id=BWHE6KLWXYR4G44A2U7WZV734XZFMIMHOUDNBVAZ4WS3XJI3XGZQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610031867 (1h39m47.390965248s) [noise=true , address=104.36.82.7:4912 , piece_id=3Y5QPXF2KO3RSR35UGUR5JIMNRFNUJ4FEHPE2I4QD4HLDGVEGTRQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610254432 (1h39m47.391187817s) [noise=true , address=104.36.82.15:4259 , piece_id=GEZGFRDTU2DM4IPPUIRYSGIRKBUWY5O3AZR4SFKDL3CA5OKYVHYQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610538752 (1h39m47.39147214s) [noise=true , address=104.36.82.13:4185 , piece_id=BS2WXJ2N6ROPR26T3FH77UMGOSJEJ2T2KQV3Y7CV2VU4MDNYWQXQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610641632 (1h39m47.391575018s) [noise=true , address=45.91.116.6:4974 , piece_id=JFAOMJOYSEPVXY2L37JHGXH4QFGW25WFJWU7SGKAF74FUVU7VI6A, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.611015304 (1h39m47.391948687s) [noise=true , address=45.91.116.13:4154 , piece_id=WIU4D4KNWUYTJDU67XUI5NL6CK6R45N7PCZN3XRGZG4NNIKERMQA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.611768742 (1h39m47.392702129s) [noise=true , address=45.91.116.11:4119 , piece_id=SWSMSBOT3LYILWGTJFMJ6CAI57KBJFMMPPJO5CUUC7AIIAIHFTCQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.612180607 (1h39m47.393113995s) [noise=true , address=104.36.82.5:4839 , piece_id=U2GI3NFFYQMFJ7BQKUUD4AYJ2IRSUBNBOLYXUZMD7UPUBJYOFFKQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.612571320 (1h39m47.393504708s) [noise=true , address=104.36.82.11:4115 , piece_id=WFALXWKWHDQXLCDJRMVMTHMGWBKM6XSNU5K6T7EXVV72NVVPWJMQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.613078814 (1h39m47.394012201s) [noise=true , address=45.91.116.14:4189 , piece_id=6CIYF3QPA5LTUKN7Q2HCRX3HBU52OYX23K5ILWFG5VYVHQPJXFBQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.613860849 (1h39m47.394794237s) [noise=true , address=77.90.31.19:4841 , piece_id=WFVZIDNWXKGUWXSVHVFKNWJYVEDIYRGJFS5YDGNKZD6744XAU7MQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.614410419 (1h39m47.395343811s) [noise=true , address=45.91.116.5:4298 , piece_id=H3N45NWUCS2TAAUTGN5O342MN5VTWQM4V4SM5FPZ2QEBY5ZCTUIA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.614924411 (1h39m47.395857799s) [noise=true , address=45.91.116.10:4081 , piece_id=7IWRBV5ZTO2C5CVTQCRCTTXXPJNPXL2KQWAJJL4TLGO76R2N3ZRQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.724500405 (1h39m47.505433795s) [upload=1529]: Next returned nil segment Jul 26 23:31:31.724525907 (1h39m47.5054593s) [upload=1529]: Waiting for error group managing segments... Jul 26 23:31:31.733385014 (1h39m47.514318396s) [noise=true , address=45.91.116.13:4154 , piece_id=WIU4D4KNWUYTJDU67XUI5NL6CK6R45N7PCZN3XRGZG4NNIKERMQA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.736702712 (1h39m47.517636096s) [noise=true , address=45.91.116.10:4081 , piece_id=7IWRBV5ZTO2C5CVTQCRCTTXXPJNPXL2KQWAJJL4TLGO76R2N3ZRQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.736898929 (1h39m47.5178323s) [noise=true , address=45.91.116.6:4974 , piece_id=JFAOMJOYSEPVXY2L37JHGXH4QFGW25WFJWU7SGKAF74FUVU7VI6A, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.746132211 (1h39m47.52706558s) [noise=true , address=77.90.31.15:4698 , piece_id=7ZYN4RXVVEI6BS2JW4IVER7QNYF6JV6SZY3JREKKZUYBNVHQIR6Q, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.781376503 (1h39m47.562309873s) [noise=true , address=45.91.116.11:4119 , piece_id=SWSMSBOT3LYILWGTJFMJ6CAI57KBJFMMPPJO5CUUC7AIIAIHFTCQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.826326868 (1h39m47.607260238s) [noise=true , address=45.91.116.14:4189 , piece_id=6CIYF3QPA5LTUKN7Q2HCRX3HBU52OYX23K5ILWFG5VYVHQPJXFBQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.966957645 (1h39m47.747891023s) [noise=true , address=45.91.116.5:4298 , piece_id=H3N45NWUCS2TAAUTGN5O342MN5VTWQM4V4SM5FPZ2QEBY5ZCTUIA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:32.179086815 (1h39m47.960020198s) [noise=true , address=45.91.116.9:4263 , piece_id=BWHE6KLWXYR4G44A2U7WZV734XZFMIMHOUDNBVAZ4WS3XJI3XGZQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:33.122222841 (1h39m48.90315622s) [noise=true , address=77.90.31.19:4841 , piece_id=WFVZIDNWXKGUWXSVHVFKNWJYVEDIYRGJFS5YDGNKZD6744XAU7MQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:32:01.514610852 (1h40m17.295544215s) [noise=true , address=104.36.82.11:4115 , piece_id=WFALXWKWHDQXLCDJRMVMTHMGWBKM6XSNU5K6T7EXVV72NVVPWJMQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:56142->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.514704470 (1h40m17.295637842s) [noise=true , address=104.36.82.5:4839 , piece_id=U2GI3NFFYQMFJ7BQKUUD4AYJ2IRSUBNBOLYXUZMD7UPUBJYOFFKQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:50444->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.514767860 (1h40m17.295701222s) [noise=true , address=104.36.82.7:4912 , piece_id=3Y5QPXF2KO3RSR35UGUR5JIMNRFNUJ4FEHPE2I4QD4HLDGVEGTRQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.514953540 (1h40m17.295886921s) [noise=true , address=104.36.82.12:4149 , piece_id=PTPZSM423OKVFTTZBXYZBKYLEU72AQY55QBL2SE5CRRH7QKYHFDA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515171579 (1h40m17.296104942s) [noise=true , address=104.36.82.15:4259 , piece_id=GEZGFRDTU2DM4IPPUIRYSGIRKBUWY5O3AZR4SFKDL3CA5OKYVHYQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515298907 (1h40m17.296232272s) [noise=true , address=104.36.82.13:4185 , piece_id=BS2WXJ2N6ROPR26T3FH77UMGOSJEJ2T2KQV3Y7CV2VU4MDNYWQXQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515353693 (1h40m17.296287056s) [noise=true , address=77.90.31.18:4805 , piece_id=3Q4F7FHNVURJIXDHTXU55ACLQENASWHGE4RSY744USTVD5DRXBUA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 159.223.143.39:47822->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515375244 (1h40m17.296308622s) [seg_pos={0 0}, upload=1529]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:32:01.550559865 (1h40m17.331493233s) [upload=1529]: Done uploading segments... Jul 26 23:32:01.550603266 (1h40m17.331536643s) [upload=1529]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.04319252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.025842406s FAIL (2025-07-26 23:31:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== eces (needed at least 14 but got 12) Jul 26 23:30:53.234981356 (1h39m9.015914735s) [upload=1523]: Starting upload Jul 26 23:30:53.235103174 (1h39m9.01603656s) [upload=1523]: Done starting upload Jul 26 23:30:53.235194854 (1h39m9.016128219s) [upload=1523]: Uploading segments... Jul 26 23:30:53.235479853 (1h39m9.016413231s) [upload=1523]: Got next segment. Inline: false Jul 26 23:30:53.235544266 (1h39m9.016477645s) [upload=1523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:30:53.235556593 (1h39m9.016489969s) [upload=1523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:30:53.280130951 (1h39m9.061064336s) [seg_pos={0 0}, upload=1523]: Begin upload segment... Jul 26 23:30:53.280213847 (1h39m9.061147232s) [seg_pos={0 0}, upload=1523]: Done begin upload segment. Jul 26 23:30:53.280505427 (1h39m9.061438799s) [noise=true , address=104.36.82.20:4448 , piece_id=JZ74YDQEPTPKI3C4LP5NUUXKZ47DPMYCJX6LHN7ZPLZEAGC37KVA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.280565590 (1h39m9.061498972s) [noise=true , address=79.127.201.215:4355 , piece_id=MUXZIWTKSFNHRHCUALO7AD4ZJLZLUSDTVR5D7YVHLGU7AQZ2ILDQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.282010791 (1h39m9.062944305s) [noise=true , address=77.90.31.16:4736 , piece_id=JUC62XSYGC37MYPVAVIBAMI3EAUJ3QZR6RMEPK5S3ZL7DTPMED7Q, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.282579452 (1h39m9.063512816s) [noise=true , address=79.127.219.51:4210 , piece_id=IVKGFJ6IUIX36ADNH5OTAIUPZ6GZPQBOPZOQALVRZZ4R6MZBPWSQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.283594557 (1h39m9.064527926s) [noise=true , address=79.127.201.215:4354 , piece_id=BGSFXPMSS6PYPGMWBUFTHBDPKJAW3AH4SN42Z7QQLBWGI5I2PPZQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.283810763 (1h39m9.064744138s) [noise=true , address=79.127.205.246:4287 , piece_id=WJUMJZLHONESPZT3Z2CLH57OL4FCCZRV74NB5SAE237OI3W64DRA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.284389366 (1h39m9.065322733s) [noise=true , address=64.49.13.140:4508 , piece_id=N3VC4AINDPE7PZT75AF7DHZTKYUUDPBRNE4LCOIFXN7HX5SGOQXQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.284562838 (1h39m9.06549621s) [noise=true , address=79.127.205.235:4675 , piece_id=2AKE26X7XTKCBXIXQF2ZQBRCL2CVGEKZMLDA2TRZFUITN54ZJOPA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.285085172 (1h39m9.066018541s) [noise=true , address=45.91.116.14:4194 , piece_id=5G3IQOILJAPO7KLHLNYPYBO3ZLNDI52643MTPZQN3GIC3W4DD4IQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.285422117 (1h39m9.066355486s) [noise=true , address=104.36.82.55:4221 , piece_id=ALTQAIV6CSRXASRCS3UNS36ASMQTVZUTO4O2OQXAXSU5XWKNPKIQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.285757431 (1h39m9.0666908s) [noise=true , address=79.127.205.234:4609 , piece_id=JGGVMCED4XAQYTNYE6TTXSZG72GOOGW73PW35FHTCKLT4NQFRFIQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.286067371 (1h39m9.067000742s) [noise=true , address=104.36.82.53:4128 , piece_id=TLSTW52JLUKIKNS67NJCB2EH4V3Y24QJ6OV6VB5BU4WUXWWFJN7Q, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.286386974 (1h39m9.067320345s) [noise=true , address=79.127.205.245:4225 , piece_id=KA22FTY3LYQSL4HNXBIELAD37LEH55UVHYGRQ2T3KTJB7KC5HUQQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.286669012 (1h39m9.06760238s) [noise=true , address=79.127.219.47:4048 , piece_id=MTP7ON4CLWIU6JGKVINATUPBKBQPC7FSRNWNVSBYDYRB3RQDYBYA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.287043911 (1h39m9.067977281s) [noise=true , address=77.90.31.15:4724 , piece_id=VSNW47ZUPIY3ZVZYDEVDQBT6ZLCOP4PJ7HLHOI5ZFFI4MMYF2BHQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.287479008 (1h39m9.068412375s) [noise=true , address=104.36.82.54:4174 , piece_id=GWJNTC5354WDUCMIS76YECL7DNPHTJBYWWWFWN46TGFZPIHTFMPA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.287787298 (1h39m9.068720671s) [noise=true , address=104.36.82.47:4552 , piece_id=SWQABTTDRTVIBCWANZSIKWOVTY6NWF63RUQAZZAARMGDCBYJO7UQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.288151713 (1h39m9.069085093s) [noise=true , address=79.127.201.215:4349 , piece_id=2F6GEWJLCCEAW2XC4KP6U2QVT6LSIYBILTRUJUNRKLR4BVFJSMVA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.372316009 (1h39m9.153249371s) [upload=1523]: Next returned nil segment Jul 26 23:30:53.372341457 (1h39m9.153274833s) [upload=1523]: Waiting for error group managing segments... Jul 26 23:30:53.413241358 (1h39m9.194174739s) [noise=true , address=77.90.31.15:4724 , piece_id=VSNW47ZUPIY3ZVZYDEVDQBT6ZLCOP4PJ7HLHOI5ZFFI4MMYF2BHQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.626257370 (1h39m9.407190733s) [noise=true , address=79.127.219.51:4210 , piece_id=IVKGFJ6IUIX36ADNH5OTAIUPZ6GZPQBOPZOQALVRZZ4R6MZBPWSQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.874888347 (1h39m9.655821716s) [noise=true , address=79.127.205.235:4675 , piece_id=2AKE26X7XTKCBXIXQF2ZQBRCL2CVGEKZMLDA2TRZFUITN54ZJOPA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.890001423 (1h39m9.670934792s) [noise=true , address=79.127.205.245:4225 , piece_id=KA22FTY3LYQSL4HNXBIELAD37LEH55UVHYGRQ2T3KTJB7KC5HUQQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.907214248 (1h39m9.688147614s) [noise=true , address=79.127.219.47:4048 , piece_id=MTP7ON4CLWIU6JGKVINATUPBKBQPC7FSRNWNVSBYDYRB3RQDYBYA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.949228336 (1h39m9.73016172s) [noise=true , address=79.127.205.246:4287 , piece_id=WJUMJZLHONESPZT3Z2CLH57OL4FCCZRV74NB5SAE237OI3W64DRA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:54.824171521 (1h39m10.60510489s) [noise=true , address=79.127.205.234:4609 , piece_id=JGGVMCED4XAQYTNYE6TTXSZG72GOOGW73PW35FHTCKLT4NQFRFIQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:58.994629630 (1h39m14.775562993s) [noise=true , address=45.91.116.14:4194 , piece_id=5G3IQOILJAPO7KLHLNYPYBO3ZLNDI52643MTPZQN3GIC3W4DD4IQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:00.298638954 (1h39m16.079572325s) [noise=true , address=64.49.13.140:4508 , piece_id=N3VC4AINDPE7PZT75AF7DHZTKYUUDPBRNE4LCOIFXN7HX5SGOQXQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:09.244670085 (1h39m25.025603452s) [noise=true , address=79.127.201.215:4355 , piece_id=MUXZIWTKSFNHRHCUALO7AD4ZJLZLUSDTVR5D7YVHLGU7AQZ2ILDQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:15.992987059 (1h39m31.773920439s) [noise=true , address=77.90.31.16:4736 , piece_id=JUC62XSYGC37MYPVAVIBAMI3EAUJ3QZR6RMEPK5S3ZL7DTPMED7Q, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:30.443778609 (1h39m46.224711971s) [noise=true , address=79.127.201.215:4354 , piece_id=BGSFXPMSS6PYPGMWBUFTHBDPKJAW3AH4SN42Z7QQLBWGI5I2PPZQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:46.506777124 (1h40m2.287710501s) [noise=true , address=79.127.201.215:4349 , piece_id=2F6GEWJLCCEAW2XC4KP6U2QVT6LSIYBILTRUJUNRKLR4BVFJSMVA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:53.198614381 (1h40m8.979547762s) [noise=true , address=104.36.82.55:4221 , piece_id=ALTQAIV6CSRXASRCS3UNS36ASMQTVZUTO4O2OQXAXSU5XWKNPKIQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: rpc: dial tcp 104.36.82.55:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198696025 (1h40m8.979629391s) [noise=true , address=104.36.82.54:4174 , piece_id=GWJNTC5354WDUCMIS76YECL7DNPHTJBYWWWFWN46TGFZPIHTFMPA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: rpc: dial tcp 104.36.82.54:4174: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198755921 (1h40m8.979689287s) [noise=true , address=104.36.82.47:4552 , piece_id=SWQABTTDRTVIBCWANZSIKWOVTY6NWF63RUQAZZAARMGDCBYJO7UQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198790524 (1h40m8.979723901s) [noise=true , address=104.36.82.20:4448 , piece_id=JZ74YDQEPTPKI3C4LP5NUUXKZ47DPMYCJX6LHN7ZPLZEAGC37KVA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp, address:104.36.82.20:4448): protocol: rpc: dial tcp 104.36.82.20:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198937385 (1h40m8.97987077s) [noise=true , address=104.36.82.53:4128 , piece_id=TLSTW52JLUKIKNS67NJCB2EH4V3Y24QJ6OV6VB5BU4WUXWWFJN7Q, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 159.223.143.39:35398->104.36.82.53:4128: use of closed network connection; conn closed: op Jul 26 23:31:53.199013779 (1h40m8.979947202s) [seg_pos={0 0}, upload=1523]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:31:53.222144113 (1h40m9.003077491s) [upload=1523]: Done uploading segments... Jul 26 23:31:53.222175757 (1h40m9.003109158s) [upload=1523]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.025865114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XXX.X....XX.X.X.X.......XXX...X] duration: 30.001016143s FAIL (2025-07-26 23:31:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001104709s edge gateway us-select-1

1
...
6762
6763
6764
6765
6766
...
7257