Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 362824 reports (Page 6765 of 7257)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................XXX.X....XX.X.X.X.......XXX] duration: 30.001469488s FAIL (2025-07-26 23:28:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001495926s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..........X.X..........XX...X...X..XX.X...X..X.......X...X..] duration: 48.784177569s FAIL (2025-07-26 23:28:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.784222132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X........................X......XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031097825s FAIL (2025-07-26 23:27:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.031133388s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031851213s FAIL (2025-07-26 23:27:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== aa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327916538 (1h34m47.10884992s) [seg_pos={0 0}, upload=1450]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:26:31.354126428 (1h34m47.135059805s) [upload=1450]: Done uploading segments... Jul 26 23:26:31.354156676 (1h34m47.135090053s) [upload=1450]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:27:12.406126641 (1h35m28.187060017s) [upload=1467]: Starting upload Jul 26 23:27:12.406195802 (1h35m28.187129168s) [upload=1467]: Done starting upload Jul 26 23:27:12.406422274 (1h35m28.187355641s) [upload=1467]: Uploading segments... Jul 26 23:27:12.406611296 (1h35m28.187544675s) [upload=1467]: Got next segment. Inline: false Jul 26 23:27:12.406742913 (1h35m28.18767629s) [upload=1467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:27:12.406751844 (1h35m28.187685223s) [upload=1467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:27:12.435780043 (1h35m28.216713419s) [seg_pos={0 0}, upload=1467]: Begin upload segment... Jul 26 23:27:12.435828004 (1h35m28.216761373s) [seg_pos={0 0}, upload=1467]: Done begin upload segment. Jul 26 23:27:12.435859205 (1h35m28.216792579s) [noise=true , address=77.90.31.16:4734 , piece_id=IWMWDP6F4C6N3XE45DFKCETMEIM27T3WFNDIZ4WJ5WBJPJ2EEBDQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.435894044 (1h35m28.216827429s) [noise=true , address=45.91.116.7:4011 , piece_id=Y5AASEQYGOGYPSFVOIK6ZPXGKFDDF2UJ7UZLICXIJRDPV5MMVPQQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.436391448 (1h35m28.217324831s) [noise=true , address=45.91.116.10:4083 , piece_id=EFYVD2SLMPSHHQJRGNT3VKMKQQZV6AW4JX7FC6KQISBOUNRILH5Q, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.436558578 (1h35m28.217491964s) [noise=true , address=77.90.31.12:4590 , piece_id=SXRPOVG2CDNAJACM6SOSBNW42TMLXPC5ZLJ3FAEEGG6PMPEQKMXQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.436786676 (1h35m28.21772006s) [noise=true , address=45.91.116.5:4298 , piece_id=IGDSSBE6WGYYUYMQQ5U66P5FI3LQ7JANJOWZJR2SMUTYMLIF4M7Q, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437151469 (1h35m28.218084847s) [noise=true , address=104.36.82.5:4839 , piece_id=RZNCJ7M2X7VJDJWJBRLG4GE24J4OZ2JEVWJ4YMZBCYBQRDKZPZTA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437484548 (1h35m28.218417927s) [noise=true , address=45.91.116.13:4153 , piece_id=ANER425M4SOCJUMQEWD742OF4JZJUOLEYP6HHPBR6LFH5MSKXBQA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437820365 (1h35m28.218753743s) [noise=true , address=104.36.82.4:4801 , piece_id=CM7DVF77TXSIWLSLQLZ56FUGXKVB2GNO7GLAMYSTD6PPDO6BS2DQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437932378 (1h35m28.218865764s) [noise=true , address=45.91.116.12:4225 , piece_id=FJTWSVHCWCO4CP5NOZI6A7LF5K2ZLBJQDCE2XNUYQXMDZCXG55SA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.438370932 (1h35m28.219304321s) [noise=true , address=104.36.82.10:4079 , piece_id=Q4PVG5RZNVXHE7AWTNMNJJLYFAPHZCO7IUPIDCPJDXVBKQK4GJCA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.438793254 (1h35m28.219726639s) [noise=true , address=104.36.82.13:4186 , piece_id=2Q62YQEDO3RD52IP72JQMMR7ZBANB4VOWSHGPI32KCPHDN6VDHCA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.439124793 (1h35m28.220058176s) [noise=true , address=45.91.116.6:4975 , piece_id=IFV5TOWWFA7TQ3ZAGPKJRJYL44CV3USE5KPTDR2RU22HWNC5EKBQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.439515696 (1h35m28.220449074s) [noise=true , address=104.36.82.9:4005 , piece_id=3GYIPBREPTS3H7OVRC4QLG2EKXJFCDGPBLEQYGTDMU4GOVLXARFQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.439912524 (1h35m28.220845903s) [noise=true , address=45.91.116.8:4048 , piece_id=PSJ2DXQO25FDASVOV33ZNOUMVZTXINTVUWOCOTRZWPSVW722B2IQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.440057808 (1h35m28.220991194s) [noise=true , address=45.91.116.12:4226 , piece_id=WUSEEGGCESNRVCX7QOQNMFKVJTCAO2XNAH22VFMAZOS4BKW6CBJA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.440587514 (1h35m28.221520964s) [noise=true , address=45.91.116.10:4081 , piece_id=OQNVHOOVR7K2Q5U7SNU3SDJMEJDFINCAF67YOYFJJA6XHTBUK6BQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.546057090 (1h35m28.32699049s) [upload=1467]: Next returned nil segment Jul 26 23:27:12.546098741 (1h35m28.327032119s) [upload=1467]: Waiting for error group managing segments... Jul 26 23:27:12.548967636 (1h35m28.329901014s) [noise=true , address=45.91.116.10:4081 , piece_id=OQNVHOOVR7K2Q5U7SNU3SDJMEJDFINCAF67YOYFJJA6XHTBUK6BQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.549616717 (1h35m28.330550095s) [noise=true , address=45.91.116.13:4153 , piece_id=ANER425M4SOCJUMQEWD742OF4JZJUOLEYP6HHPBR6LFH5MSKXBQA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.550171041 (1h35m28.331104411s) [noise=true , address=45.91.116.7:4011 , piece_id=Y5AASEQYGOGYPSFVOIK6ZPXGKFDDF2UJ7UZLICXIJRDPV5MMVPQQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.550732851 (1h35m28.331666218s) [noise=true , address=45.91.116.12:4225 , piece_id=FJTWSVHCWCO4CP5NOZI6A7LF5K2ZLBJQDCE2XNUYQXMDZCXG55SA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.551402957 (1h35m28.332336323s) [noise=true , address=45.91.116.6:4975 , piece_id=IFV5TOWWFA7TQ3ZAGPKJRJYL44CV3USE5KPTDR2RU22HWNC5EKBQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.552146469 (1h35m28.333079845s) [noise=true , address=45.91.116.12:4226 , piece_id=WUSEEGGCESNRVCX7QOQNMFKVJTCAO2XNAH22VFMAZOS4BKW6CBJA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.552711637 (1h35m28.333645002s) [noise=true , address=45.91.116.10:4083 , piece_id=EFYVD2SLMPSHHQJRGNT3VKMKQQZV6AW4JX7FC6KQISBOUNRILH5Q, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.553390254 (1h35m28.334323618s) [noise=true , address=45.91.116.8:4048 , piece_id=PSJ2DXQO25FDASVOV33ZNOUMVZTXINTVUWOCOTRZWPSVW722B2IQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.980926756 (1h35m28.761860133s) [noise=true , address=45.91.116.5:4298 , piece_id=IGDSSBE6WGYYUYMQQ5U66P5FI3LQ7JANJOWZJR2SMUTYMLIF4M7Q, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:13.950279768 (1h35m29.731213138s) [noise=true , address=77.90.31.12:4590 , piece_id=SXRPOVG2CDNAJACM6SOSBNW42TMLXPC5ZLJ3FAEEGG6PMPEQKMXQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:42.373490982 (1h35m58.154424358s) [noise=true , address=77.90.31.16:4734 , piece_id=IWMWDP6F4C6N3XE45DFKCETMEIM27T3WFNDIZ4WJ5WBJPJ2EEBDQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373560747 (1h35m58.154494123s) [noise=true , address=104.36.82.13:4186 , piece_id=2Q62YQEDO3RD52IP72JQMMR7ZBANB4VOWSHGPI32KCPHDN6VDHCA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:53166->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373675318 (1h35m58.154608694s) [noise=true , address=104.36.82.10:4079 , piece_id=Q4PVG5RZNVXHE7AWTNMNJJLYFAPHZCO7IUPIDCPJDXVBKQK4GJCA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:42.373712909 (1h35m58.154646281s) [noise=true , address=104.36.82.4:4801 , piece_id=CM7DVF77TXSIWLSLQLZ56FUGXKVB2GNO7GLAMYSTD6PPDO6BS2DQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:51702->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373758162 (1h35m58.154691523s) [noise=true , address=104.36.82.9:4005 , piece_id=3GYIPBREPTS3H7OVRC4QLG2EKXJFCDGPBLEQYGTDMU4GOVLXARFQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373788016 (1h35m58.154721391s) [noise=true , address=104.36.82.5:4839 , piece_id=RZNCJ7M2X7VJDJWJBRLG4GE24J4OZ2JEVWJ4YMZBCYBQRDKZPZTA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:57326->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373838032 (1h35m58.154771411s) [seg_pos={0 0}, upload=1467]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:27:42.403337269 (1h35m58.184270635s) [upload=1467]: Done uploading segments... Jul 26 23:27:42.403363901 (1h35m58.184297279s) [upload=1467]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.031878197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035368223s FAIL (2025-07-26 23:27:39 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ... Jul 26 23:26:39.659234999 (1h34m55.440168374s) [noise=true , address=104.36.82.55:4202 , piece_id=YPXNX3S5PSYK5DRNZYKWMRIHEAFP7FCLBNDTNAJJ27MYPGIIGRFQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659344193 (1h34m55.440277572s) [noise=true , address=64.49.13.154:4927 , piece_id=6Z3LSXHIX3JXS2LZCQQ3ABUOR55LEQ3OKVYIIDYOOO53KI7GLELQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659481433 (1h34m55.440414807s) [noise=true , address=104.36.82.54:4189 , piece_id=PK7G3SJ7UEUNRXJIEFIKCXUQSDESHJL372Z7LYITW7I42MXUAZKA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659732007 (1h34m55.440665385s) [noise=true , address=104.36.82.52:4111 , piece_id=UAUCOWACA6QYBBM7W3TIZFJ3W4SFHCLXWXCWZS5L6TOJBCRPUOZQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659747536 (1h34m55.44068091s) [noise=true , address=77.90.31.15:4712 , piece_id=UFKXB7ETSKH32EXYABNBAINZSPBW5A5MFZYEVFTOG5CE2SF67GDQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659786218 (1h34m55.440719593s) [noise=true , address=104.36.82.47:4527 , piece_id=MK524VHV57PBATOQH7GW4MUOFCCWNUNP2PYH5UBE64XOTZLUZLFA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660125196 (1h34m55.441058574s) [noise=true , address=77.90.31.17:4786 , piece_id=YHNOCPEHXANRHEEBOD4KWD7FZ2Z2BYZC2DE5Q5XHZXYA6F43KRSQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660286927 (1h34m55.441220311s) [noise=true , address=79.127.201.215:4346 , piece_id=2XR6CUELXRNQEHTXTII54HES7VPEKIK77WKQRKZQBYBTEHQ7HAKA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660472628 (1h34m55.441406007s) [noise=true , address=79.127.219.38:4006 , piece_id=BQTVRTTKNOREACEFSYARXR3SHKBVBE3ICZAOBMAYHOPCBA464EFA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660652734 (1h34m55.441586112s) [noise=true , address=79.127.205.246:4280 , piece_id=XGJ5PFEESTFNKRRQRBOF2FBGB3YWZ7CSO7MVOKIBIBSJWYH4ZQYQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660790181 (1h34m55.441723562s) [noise=true , address=156.146.43.225:4448 , piece_id=GV4BCGDA4X6IG47A7POYPRPP3IIGCAXO34DZDJMVJDIVY625Z3FA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.661118495 (1h34m55.44205187s) [noise=true , address=104.36.82.56:4241 , piece_id=6ZEUOUVT7HE66BO65BMGZR7TASSPIRT6O3PNDNTBDPD33AIMZ6SQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.661722162 (1h34m55.44265553s) [noise=true , address=104.36.82.12:4154 , piece_id=JSQ3DG4P4OMLM3WGADASSV2FBULSRBRRQ6D63FOZ723Y2AG2UKBQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.664110323 (1h34m55.445043698s) [noise=true , address=77.90.31.19:4844 , piece_id=B7YNOT7HXKFTYYCPTJ5ZAZYEDTFF745V6TMT7GOLAV6IJSRQQBVQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.697218787 (1h34m55.478152164s) [noise=true , address=77.90.31.13:4634 , piece_id=DUPUJDZPB3Y45BFV3ARYZCGEMID7SI5GE2YABVBQARKD6O2I2YJQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, address:77.90.31.13:4634): protocol: storage node overloaded, request limit: 2000 Jul 26 23:26:39.736192088 (1h34m55.517125462s) [upload=1460]: Next returned nil segment Jul 26 23:26:39.736209892 (1h34m55.517143259s) [upload=1460]: Waiting for error group managing segments... Jul 26 23:26:39.740390866 (1h34m55.521324233s) [noise=true , address=79.127.219.38:4006 , piece_id=BQTVRTTKNOREACEFSYARXR3SHKBVBE3ICZAOBMAYHOPCBA464EFA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:39.777614590 (1h34m55.558547952s) [noise=true , address=77.90.31.17:4786 , piece_id=YHNOCPEHXANRHEEBOD4KWD7FZ2Z2BYZC2DE5Q5XHZXYA6F43KRSQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:40.631541532 (1h34m56.412474916s) [noise=true , address=79.127.205.246:4280 , piece_id=XGJ5PFEESTFNKRRQRBOF2FBGB3YWZ7CSO7MVOKIBIBSJWYH4ZQYQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:41.210645432 (1h34m56.991578812s) [noise=true , address=156.146.43.225:4448 , piece_id=GV4BCGDA4X6IG47A7POYPRPP3IIGCAXO34DZDJMVJDIVY625Z3FA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:47.410738393 (1h35m3.191671763s) [noise=true , address=64.49.13.154:4927 , piece_id=6Z3LSXHIX3JXS2LZCQQ3ABUOR55LEQ3OKVYIIDYOOO53KI7GLELQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:50.571174910 (1h35m6.35210828s) [noise=true , address=79.127.201.215:4346 , piece_id=2XR6CUELXRNQEHTXTII54HES7VPEKIK77WKQRKZQBYBTEHQ7HAKA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:27:09.836278592 (1h35m25.61721196s) [noise=true , address=77.90.31.19:4844 , piece_id=B7YNOT7HXKFTYYCPTJ5ZAZYEDTFF745V6TMT7GOLAV6IJSRQQBVQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:27:39.587129956 (1h35m55.368063332s) [noise=true , address=77.90.31.15:4712 , piece_id=UFKXB7ETSKH32EXYABNBAINZSPBW5A5MFZYEVFTOG5CE2SF67GDQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:27:39.587435246 (1h35m55.368368621s) [noise=true , address=104.36.82.54:4189 , piece_id=PK7G3SJ7UEUNRXJIEFIKCXUQSDESHJL372Z7LYITW7I42MXUAZKA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: write tcp 159.223.143.39:44002->104.36.82.54:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587539095 (1h35m55.36847247s) [noise=true , address=104.36.82.56:4241 , piece_id=6ZEUOUVT7HE66BO65BMGZR7TASSPIRT6O3PNDNTBDPD33AIMZ6SQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: write tcp 159.223.143.39:60448->104.36.82.56:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587595891 (1h35m55.368529268s) [noise=true , address=104.36.82.35:4605 , piece_id=23EFVOTXIL52TEHUBQHTQPI4TMMVLWETGHGUCKO37MHBCQNZYANA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:104.36.82.35:4605): protocol: write tcp 159.223.143.39:43762->104.36.82.35:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587704546 (1h35m55.368637911s) [noise=true , address=104.36.82.52:4111 , piece_id=UAUCOWACA6QYBBM7W3TIZFJ3W4SFHCLXWXCWZS5L6TOJBCRPUOZQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 159.223.143.39:55594->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587821522 (1h35m55.36875489s) [noise=true , address=104.36.82.12:4154 , piece_id=JSQ3DG4P4OMLM3WGADASSV2FBULSRBRRQ6D63FOZ723Y2AG2UKBQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: write tcp 159.223.143.39:40554->104.36.82.12:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587897400 (1h35m55.368830765s) [noise=true , address=104.36.82.53:4134 , piece_id=LSDNUBJQ2B6VKKMSRNF77PXKE4PEVUS7GZSJ5KWWP3X3RLE3FISA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 159.223.143.39:47844->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587989813 (1h35m55.368923178s) [noise=true , address=104.36.82.46:4488 , piece_id=ZALKVH3EFKB7ZVKJYKLKBKF444KRAQR4ID3PRMMA4XCS674CCKGA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: write tcp 159.223.143.39:40048->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.588144771 (1h35m55.369078149s) [noise=true , address=104.36.82.55:4202 , piece_id=YPXNX3S5PSYK5DRNZYKWMRIHEAFP7FCLBNDTNAJJ27MYPGIIGRFQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: write tcp 159.223.143.39:42452->104.36.82.55:4202: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.588218463 (1h35m55.369151839s) [noise=true , address=104.36.82.47:4527 , piece_id=MK524VHV57PBATOQH7GW4MUOFCCWNUNP2PYH5UBE64XOTZLUZLFA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: write tcp 159.223.143.39:39052->104.36.82.47:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.588332362 (1h35m55.369265747s) [seg_pos={0 0}, upload=1460]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:27:39.620797969 (1h35m55.401731345s) [upload=1460]: Done uploading segments... Jul 26 23:27:39.620828459 (1h35m55.401761838s) [upload=1460]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.035395033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................................XXX.X....XX.X.X.X.......XX] duration: 30.000944629s FAIL (2025-07-26 23:27:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00097002s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [...X........................X......XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.053848936s FAIL (2025-07-26 23:27:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.053875618s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................................XXX.X....XX.X.X.X.......X] duration: 30.001902058s FAIL (2025-07-26 23:26:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001926226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X........................X......XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031751079s FAIL (2025-07-26 23:26:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.031811398s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.029376822s FAIL (2025-07-26 23:26:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 43 (1h34m17.151802022s) [upload=1450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:26:01.420285671 (1h34m17.201219062s) [seg_pos={0 0}, upload=1450]: Begin upload segment... Jul 26 23:26:01.420356758 (1h34m17.201290127s) [seg_pos={0 0}, upload=1450]: Done begin upload segment. Jul 26 23:26:01.420395583 (1h34m17.201328968s) [noise=true , address=45.91.116.5:4297 , piece_id=DGHNUNGANOGH7BQ7NAXOJBJJHWZ4NU3IEILVRT7ZLX6YBBXHYQVA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.420431859 (1h34m17.201365247s) [noise=true , address=104.36.82.9:4007 , piece_id=JNQ3VJURTSKTJKVEXQD6PS5SFXTA527T6UBMXLLGEX56C6X4KHOA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.420935417 (1h34m17.201868797s) [noise=true , address=77.90.31.14:4661 , piece_id=ALF43VKH43T3ONH475BERD4ACWJ4V5VBJSMBS2QQMMV7MYFGBYJA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421103557 (1h34m17.202036941s) [noise=true , address=45.91.116.14:4191 , piece_id=JWWSEQAZUFPSKP4WJGEHGXZQYDO2IL5IBUISFIFYYYABNXLE6WJQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421339083 (1h34m17.202272472s) [noise=true , address=104.36.82.13:4187 , piece_id=RY367W5IWYZWMG36WHF5VVWT3JIGX6GFUDPNB4WAFTVCDDIXL7WQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421621978 (1h34m17.202555376s) [noise=true , address=104.36.82.5:4840 , piece_id=RGMWONKFDOHRKNMXBTV2FIRTXL2GB4O2KHAEFQIRR3SJBWL4AQFA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421777483 (1h34m17.202710865s) [noise=true , address=104.36.82.12:4150 , piece_id=TW2VWDGUHVWD3VXUYDCLCBXLVCYRZDLATZB4Y256VN2IK4Y4ZF3Q, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422159781 (1h34m17.203093162s) [noise=true , address=104.36.82.5:4837 , piece_id=XT6HGSJGLDOX72ZUILJDHCHPJZV6K3XWO6OLWRDDBCMSY2ABUSBQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422211867 (1h34m17.203145254s) [noise=true , address=104.36.82.16:4295 , piece_id=ZXQL5GKPUDDYBN7DTIB4KKVEQJM4PVIFGXNJD3SWFYQS7J4B4BNA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422621888 (1h34m17.203555278s) [noise=true , address=45.91.116.8:4048 , piece_id=Z7ITEZEYW63OAFJJJN3BUCH2LOQW6FOJRBRPBUTCWRMJY3ITDCIA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422633751 (1h34m17.203567153s) [noise=true , address=104.36.82.13:4186 , piece_id=5ZWKUPNX3HMM3KKFTL4MQZROL4QLLQY35IU4Y2V2GFPNNJ6DK6HA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422706245 (1h34m17.203639632s) [noise=true , address=104.36.82.8:4969 , piece_id=I2W3QHNJ2CZBYTPCRGGLWCYFTI7N3SVK2FMOV5PKHQ3Z7PXRSMBQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.423096422 (1h34m17.204029798s) [noise=true , address=45.91.116.10:4082 , piece_id=XFQTL7NY3UNWTPVJTNIIHKEDPQPVNCCK2HHUJGEKXYGO6XCSQICQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.423146574 (1h34m17.204079954s) [noise=true , address=104.36.82.7:4912 , piece_id=3BTOM6OJUIV2YA3GXNXW2OJ274LLVTXT7P2DBGWSOZNJJCWK2OMA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.423478700 (1h34m17.204412083s) [noise=true , address=45.91.116.9:4261 , piece_id=NPKADCMCSRZGKPDXYGMBGVXPJ5GD2FQOGR5FPBLYAUEMXZLNM7RQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.436569456 (1h34m17.217502836s) [noise=true , address=104.36.82.3:4767 , piece_id=WGKWPPH7TKDSXYMR3IB4BZPE2ROBWMYTLI6DHFJKMRN7GTB3XBHQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.517386527 (1h34m17.298319894s) [upload=1450]: Next returned nil segment Jul 26 23:26:01.517403712 (1h34m17.298337079s) [upload=1450]: Waiting for error group managing segments... Jul 26 23:26:01.521040588 (1h34m17.30197397s) [noise=true , address=45.91.116.10:4082 , piece_id=XFQTL7NY3UNWTPVJTNIIHKEDPQPVNCCK2HHUJGEKXYGO6XCSQICQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.521679172 (1h34m17.302612535s) [noise=true , address=45.91.116.8:4048 , piece_id=Z7ITEZEYW63OAFJJJN3BUCH2LOQW6FOJRBRPBUTCWRMJY3ITDCIA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.524571978 (1h34m17.305505364s) [noise=true , address=77.90.31.14:4661 , piece_id=ALF43VKH43T3ONH475BERD4ACWJ4V5VBJSMBS2QQMMV7MYFGBYJA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.551563819 (1h34m17.332497201s) [noise=true , address=45.91.116.14:4191 , piece_id=JWWSEQAZUFPSKP4WJGEHGXZQYDO2IL5IBUISFIFYYYABNXLE6WJQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.644260327 (1h34m17.425193705s) [noise=true , address=45.91.116.5:4297 , piece_id=DGHNUNGANOGH7BQ7NAXOJBJJHWZ4NU3IEILVRT7ZLX6YBBXHYQVA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.987450406 (1h34m17.768383775s) [noise=true , address=45.91.116.9:4261 , piece_id=NPKADCMCSRZGKPDXYGMBGVXPJ5GD2FQOGR5FPBLYAUEMXZLNM7RQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:31.327252944 (1h34m47.10818631s) [noise=true , address=104.36.82.5:4837 , piece_id=XT6HGSJGLDOX72ZUILJDHCHPJZV6K3XWO6OLWRDDBCMSY2ABUSBQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:45308->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327344545 (1h34m47.108277935s) [noise=true , address=104.36.82.5:4840 , piece_id=RGMWONKFDOHRKNMXBTV2FIRTXL2GB4O2KHAEFQIRR3SJBWL4AQFA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:34572->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327415428 (1h34m47.108348803s) [noise=true , address=104.36.82.12:4150 , piece_id=TW2VWDGUHVWD3VXUYDCLCBXLVCYRZDLATZB4Y256VN2IK4Y4ZF3Q, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327467550 (1h34m47.108400926s) [noise=true , address=104.36.82.16:4295 , piece_id=ZXQL5GKPUDDYBN7DTIB4KKVEQJM4PVIFGXNJD3SWFYQS7J4B4BNA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327566254 (1h34m47.108499629s) [noise=true , address=104.36.82.9:4007 , piece_id=JNQ3VJURTSKTJKVEXQD6PS5SFXTA527T6UBMXLLGEX56C6X4KHOA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:52264->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327659406 (1h34m47.108592781s) [noise=true , address=104.36.82.13:4186 , piece_id=5ZWKUPNX3HMM3KKFTL4MQZROL4QLLQY35IU4Y2V2GFPNNJ6DK6HA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:44064->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327720925 (1h34m47.108654294s) [noise=true , address=104.36.82.7:4912 , piece_id=3BTOM6OJUIV2YA3GXNXW2OJ274LLVTXT7P2DBGWSOZNJJCWK2OMA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327799308 (1h34m47.108732672s) [noise=true , address=104.36.82.3:4767 , piece_id=WGKWPPH7TKDSXYMR3IB4BZPE2ROBWMYTLI6DHFJKMRN7GTB3XBHQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:57206->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327852056 (1h34m47.108785431s) [noise=true , address=104.36.82.13:4187 , piece_id=RY367W5IWYZWMG36WHF5VVWT3JIGX6GFUDPNB4WAFTVCDDIXL7WQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327888899 (1h34m47.108822275s) [noise=true , address=104.36.82.8:4969 , piece_id=I2W3QHNJ2CZBYTPCRGGLWCYFTI7N3SVK2FMOV5PKHQ3Z7PXRSMBQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327916538 (1h34m47.10884992s) [seg_pos={0 0}, upload=1450]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:26:31.354126428 (1h34m47.135059805s) [upload=1450]: Done uploading segments... Jul 26 23:26:31.354156676 (1h34m47.135090053s) [upload=1450]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.029407026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXX] duration: 1m0.029528898s FAIL (2025-07-26 23:26:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ding piece... Jul 26 23:25:29.039243770 (1h33m44.820177153s) [noise=true , address=45.91.116.11:4151 , piece_id=PYWIQBOK5ITSTF4UP7IQ6VMVEK5DCVTZF6NEECRDO5IOIA33HPTA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.039645110 (1h33m44.820578491s) [noise=true , address=104.36.82.47:4550 , piece_id=PA7B2GZEIV24FRGQPTA5R7XY6GGIB5T5KCA6V6JCZKBEBOBX4XKA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.039696273 (1h33m44.820629657s) [noise=true , address=79.127.205.233:4641 , piece_id=ZSIHSAHZMRRXXKBLWMTBJZ7WS6J6NYGHDIHQUG5FYRQOBVKKK6RQ, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.040124770 (1h33m44.82105815s) [noise=true , address=104.36.82.53:4136 , piece_id=6S5AVQDMSLPAYDAITWZZW64SSB26D6H6R5POJALJ54KQW6WAFUAA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.040317944 (1h33m44.82125133s) [noise=true , address=79.127.201.215:4360 , piece_id=VJJB34P7VDWX3JCT4IVX46TDVAM6KJPHDYIG4AWVAUJQ527JF3KA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.040516463 (1h33m44.821449841s) [noise=true , address=45.91.116.9:4277 , piece_id=HPYS7S2TWWLMAJDCTGO5E526EFYC4JZ6NAI7T67DNRV4WKU5PVDA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.040846198 (1h33m44.821779575s) [noise=true , address=104.36.82.5:4852 , piece_id=XMDZYLJYGQSCSKQQCIVECXWLKDZMN67KXMV2GYM3UEXDKTUN2Y2Q, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.040854419 (1h33m44.821787805s) [noise=true , address=79.127.220.101:4041 , piece_id=HDXL6BGQDKEPPBQ74DEPNM3HZME67P7H266O4MYSXJVYJRKDPSBA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.041130347 (1h33m44.822063724s) [noise=true , address=79.127.219.51:4212 , piece_id=GFRSJYL435Q3HWG6W3GU564E4ZFF3QPKFF42YBXLR7TTFRMWPWLA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.041298541 (1h33m44.822250521s) [noise=true , address=77.90.31.13:4659 , piece_id=W54H7LZC7MFNAGUC7K3MLBBVLEWWZE57S2JGTZFFAEYJ2IFZJ53A, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.041457282 (1h33m44.822390665s) [noise=true , address=79.127.205.245:4237 , piece_id=XHJ6XGFL46IA4GYF5IFZXQGMY3GWJAEWEMYE4TVLJX3EV6KQWJEA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.041759079 (1h33m44.822692458s) [noise=true , address=104.36.82.46:4506 , piece_id=LELBGNAWJQ2QVJ4RES6F525OTVLZYIBT3UEGSRYNJYWWOOVJSSZQ, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.041773291 (1h33m44.822706677s) [noise=true , address=104.36.82.56:4261 , piece_id=ZLEVZZNDMG4BNULLAAPFQ6WF7YCZSQKUXSKWFEOGPDOBISP35CGQ, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.042275055 (1h33m44.823208439s) [noise=true , address=77.90.31.16:4760 , piece_id=E7T6THWSQ5KTMW46PI4JPCUW5GYS5WFBJASGOSF2UZJ2ZTKILCMQ, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.042622615 (1h33m44.823555993s) [noise=true , address=156.146.43.225:4449 , piece_id=PZZ62FNILACYBKUVO7OA73ALHLUDDFFK3N2G4ECF2DZ7DTH7MF6A, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.042978855 (1h33m44.823912237s) [noise=true , address=104.36.82.52:4102 , piece_id=H74MD5IYMY352BJIYO7NV5M2VKSKYB4TGHM3D3A4ZA7L2NCR6JQQ, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.046243405 (1h33m44.827176773s) [noise=true , address=64.49.13.138:4482 , piece_id=ASQHDP7AR7WPKKWUYKRPEQBXLUESM7Z5WEJMHK2PWNXKOJBIODHA, seg_pos={0 0}, upload=1443]: Uploading piece... Jul 26 23:25:29.097963350 (1h33m44.878896718s) [upload=1443]: Next returned nil segment Jul 26 23:25:29.097981779 (1h33m44.878915145s) [upload=1443]: Waiting for error group managing segments... Jul 26 23:25:29.219881774 (1h33m45.000815136s) [noise=true , address=79.127.220.101:4041 , piece_id=HDXL6BGQDKEPPBQ74DEPNM3HZME67P7H266O4MYSXJVYJRKDPSBA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:29.594373126 (1h33m45.375306519s) [noise=true , address=45.91.116.9:4277 , piece_id=HPYS7S2TWWLMAJDCTGO5E526EFYC4JZ6NAI7T67DNRV4WKU5PVDA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 23:25:29.645375812 (1h33m45.426309175s) [noise=true , address=79.127.205.245:4237 , piece_id=XHJ6XGFL46IA4GYF5IFZXQGMY3GWJAEWEMYE4TVLJX3EV6KQWJEA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:29.695815394 (1h33m45.476748755s) [noise=true , address=79.127.205.233:4641 , piece_id=ZSIHSAHZMRRXXKBLWMTBJZ7WS6J6NYGHDIHQUG5FYRQOBVKKK6RQ, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:29.718232362 (1h33m45.499165741s) [noise=true , address=64.49.13.138:4482 , piece_id=ASQHDP7AR7WPKKWUYKRPEQBXLUESM7Z5WEJMHK2PWNXKOJBIODHA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:29.826121677 (1h33m45.607055043s) [noise=true , address=156.146.43.225:4449 , piece_id=PZZ62FNILACYBKUVO7OA73ALHLUDDFFK3N2G4ECF2DZ7DTH7MF6A, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:31.139319909 (1h33m46.920253288s) [noise=true , address=79.127.219.51:4212 , piece_id=GFRSJYL435Q3HWG6W3GU564E4ZFF3QPKFF42YBXLR7TTFRMWPWLA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:39.069197629 (1h33m54.850130995s) [noise=true , address=45.91.116.11:4151 , piece_id=PYWIQBOK5ITSTF4UP7IQ6VMVEK5DCVTZF6NEECRDO5IOIA33HPTA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:42.767292400 (1h33m58.548225772s) [noise=true , address=79.127.201.215:4360 , piece_id=VJJB34P7VDWX3JCT4IVX46TDVAM6KJPHDYIG4AWVAUJQ527JF3KA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:25:57.041731450 (1h34m12.822664819s) [noise=true , address=77.90.31.13:4659 , piece_id=W54H7LZC7MFNAGUC7K3MLBBVLEWWZE57S2JGTZFFAEYJ2IFZJ53A, seg_pos={0 0}, upload=1443]: Done uploading piece. err: <nil> Jul 26 23:26:28.960486180 (1h34m44.741419549s) [noise=true , address=77.90.31.16:4760 , piece_id=E7T6THWSQ5KTMW46PI4JPCUW5GYS5WFBJASGOSF2UZJ2ZTKILCMQ, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, address:77.90.31.16:4760): protocol: rpc: dial tcp 77.90.31.16:4760: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960559320 (1h34m44.74149269s) [noise=true , address=104.36.82.55:4218 , piece_id=4B5R2ZJEUBHMTQF3CSLRATXDDVMI4VP3HUYZLVK64ZUMLSXQIQQQ, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 159.223.143.39:34472->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960610622 (1h34m44.741543993s) [noise=true , address=104.36.82.47:4550 , piece_id=PA7B2GZEIV24FRGQPTA5R7XY6GGIB5T5KCA6V6JCZKBEBOBX4XKA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960687623 (1h34m44.741620991s) [noise=true , address=104.36.82.46:4506 , piece_id=LELBGNAWJQ2QVJ4RES6F525OTVLZYIBT3UEGSRYNJYWWOOVJSSZQ, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: write tcp 159.223.143.39:52294->104.36.82.46:4506: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960772042 (1h34m44.741705405s) [noise=true , address=104.36.82.56:4261 , piece_id=ZLEVZZNDMG4BNULLAAPFQ6WF7YCZSQKUXSKWFEOGPDOBISP35CGQ, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: rpc: dial tcp 104.36.82.56:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960800355 (1h34m44.741733738s) [noise=true , address=104.36.82.5:4852 , piece_id=XMDZYLJYGQSCSKQQCIVECXWLKDZMN67KXMV2GYM3UEXDKTUN2Y2Q, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:12sQvAZS92R7TTJUFjMRieYT6BP4vp2UAan4PHrhPt6m3eHPukN, address:104.36.82.5:4852): protocol: rpc: dial tcp 104.36.82.5:4852: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960840804 (1h34m44.741774171s) [noise=true , address=104.36.82.53:4136 , piece_id=6S5AVQDMSLPAYDAITWZZW64SSB26D6H6R5POJALJ54KQW6WAFUAA, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:104.36.82.53:4136): protocol: rpc: dial tcp 104.36.82.53:4136: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960877993 (1h34m44.741811365s) [noise=true , address=104.36.82.52:4102 , piece_id=H74MD5IYMY352BJIYO7NV5M2VKSKYB4TGHM3D3A4ZA7L2NCR6JQQ, seg_pos={0 0}, upload=1443]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: write tcp 159.223.143.39:37772->104.36.82.52:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:28.960915302 (1h34m44.741848691s) [seg_pos={0 0}, upload=1443]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:26:28.986880076 (1h34m44.767813448s) [upload=1443]: Done uploading segments... Jul 26 23:26:28.986929294 (1h34m44.767862676s) [upload=1443]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.029564272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...........X.X..........XX...X...X..XX.X...X..X.......X...X.] duration: 19.614580151s FAIL (2025-07-26 23:26:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.614596107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [............X.X..........XX...X...X..XX.X...X..X.......X...X] duration: 1m0.000545668s FAIL (2025-07-26 23:26:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000575871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.040245597s FAIL (2025-07-26 23:25:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ul 26 23:25:19.387641421 (1h33m35.168574797s) [seg_pos={0 0}, upload=1440]: Begin upload segment... Jul 26 23:25:19.387750205 (1h33m35.168683614s) [noise=true , address=104.36.82.5:4837 , piece_id=XDDXET5PRVZTX7MVP2HGMH6ET47TCT4RJYDLPLFCERRORJC53TXA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.388482892 (1h33m35.169416296s) [noise=true , address=104.36.82.4:4801 , piece_id=IZNZ2I6P4PPCUZOBCVGMWFSY53SDCOMMYF2B5PV3A655HJY3MN2A, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.391229330 (1h33m35.172162708s) [seg_pos={0 0}, upload=1440]: Done begin upload segment. Jul 26 23:25:19.391297109 (1h33m35.172230515s) [noise=true , address=77.90.31.16:4733 , piece_id=267J6XV6OMLLP74BJDOTFFSOLRJMVCOWJ3VT5GRBBNALRYONNLUQ, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.391810190 (1h33m35.172743582s) [noise=true , address=104.36.82.2:4729 , piece_id=LBNDS4XETZ2IQZFV3VMMGIL332TOYDUECV2E4MREDHUDR7RIU77Q, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.392410536 (1h33m35.173343921s) [noise=true , address=77.90.31.10:4519 , piece_id=FKRUPYFBB4IKY6ZQIOFJ6KTWCWTIMPKMV2HD6KE7VT6GVQMZ2DNQ, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.392792332 (1h33m35.17372571s) [noise=true , address=104.36.82.12:4151 , piece_id=E36QICHR2JJNG5JQUTTOROO75OPUWW4NZL7GP2YAYRP6WASFREHA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.393080181 (1h33m35.174013558s) [noise=true , address=45.91.116.10:4083 , piece_id=6GPNUMUK7AO4GIZX6H5QNR7BQIHEE3POVS2ATVMG2ISOCYWLI2ZA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.393415448 (1h33m35.174348832s) [noise=true , address=104.36.82.6:4875 , piece_id=IJAEG47NTFFKWVCDUPZTAMNV2Z72IQ74RILA6M7WBH2TMUULDS7Q, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.393745213 (1h33m35.17467859s) [noise=true , address=104.36.82.8:4972 , piece_id=C6SIOGACA4B3RXHWFVB6HEHDMELMIAEZT4RIOQ5DDJHDGSGCDE3A, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.394049059 (1h33m35.174982437s) [noise=true , address=104.36.82.12:4150 , piece_id=QVU6RQZGQV2AONPZ7YXARX3IO3USMVQ2SDWLU2HDZ4D453TU5KHA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.394513775 (1h33m35.175447159s) [noise=true , address=104.36.82.14:4221 , piece_id=5BRUO5QD2LWA35VTTMH66GIBPU7URG3MGCTBPT4ABYHWWY5RXQMA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.394889106 (1h33m35.175822484s) [noise=true , address=77.90.31.11:4554 , piece_id=INZXTKGEQIMD5WDJW4T2GWARHLPUVSQ3KF5OLIMZNAND4R7HL6NA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.395332488 (1h33m35.176265876s) [noise=true , address=45.91.116.13:4154 , piece_id=5PPQXFDG27MGS266YTF53A2NXTH6TXFFQYLZ7TALKQ6J6VSUQHLA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.395698988 (1h33m35.176632354s) [noise=true , address=45.91.116.5:4298 , piece_id=FOSPVCIVL4C342JQTUGO26FWRKXHSX2MFUBNYKV5ZTB2CXA4OZGQ, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.396138095 (1h33m35.177071482s) [noise=true , address=45.91.116.4:4940 , piece_id=GGF3IRKVSKHYAZYV27XA5R3JWTZMKJ5UTZGNTQAL73KVQ722WDSQ, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.396507531 (1h33m35.177440919s) [noise=true , address=77.90.31.14:4661 , piece_id=YZWGFFZF3PGT4XRDSLIUX3EMKLXYBG6XORMW762C7O44AKAVLWZA, seg_pos={0 0}, upload=1440]: Uploading piece... Jul 26 23:25:19.472345139 (1h33m35.253278504s) [upload=1440]: Next returned nil segment Jul 26 23:25:19.472362909 (1h33m35.253296292s) [upload=1440]: Waiting for error group managing segments... Jul 26 23:25:19.475645844 (1h33m35.256579208s) [noise=true , address=45.91.116.10:4083 , piece_id=6GPNUMUK7AO4GIZX6H5QNR7BQIHEE3POVS2ATVMG2ISOCYWLI2ZA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: <nil> Jul 26 23:25:19.476400596 (1h33m35.257333959s) [noise=true , address=45.91.116.4:4940 , piece_id=GGF3IRKVSKHYAZYV27XA5R3JWTZMKJ5UTZGNTQAL73KVQ722WDSQ, seg_pos={0 0}, upload=1440]: Done uploading piece. err: <nil> Jul 26 23:25:19.477690713 (1h33m35.258624083s) [noise=true , address=77.90.31.14:4661 , piece_id=YZWGFFZF3PGT4XRDSLIUX3EMKLXYBG6XORMW762C7O44AKAVLWZA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: <nil> Jul 26 23:25:19.477939981 (1h33m35.25887335s) [noise=true , address=45.91.116.13:4154 , piece_id=5PPQXFDG27MGS266YTF53A2NXTH6TXFFQYLZ7TALKQ6J6VSUQHLA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: <nil> Jul 26 23:25:19.479393901 (1h33m35.260327272s) [noise=true , address=45.91.116.5:4298 , piece_id=FOSPVCIVL4C342JQTUGO26FWRKXHSX2MFUBNYKV5ZTB2CXA4OZGQ, seg_pos={0 0}, upload=1440]: Done uploading piece. err: <nil> Jul 26 23:25:49.329201401 (1h34m5.110134789s) [noise=true , address=104.36.82.4:4801 , piece_id=IZNZ2I6P4PPCUZOBCVGMWFSY53SDCOMMYF2B5PV3A655HJY3MN2A, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.329350427 (1h34m5.110283805s) [noise=true , address=77.90.31.10:4519 , piece_id=FKRUPYFBB4IKY6ZQIOFJ6KTWCWTIMPKMV2HD6KE7VT6GVQMZ2DNQ, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.329512176 (1h34m5.110445557s) [noise=true , address=104.36.82.2:4729 , piece_id=LBNDS4XETZ2IQZFV3VMMGIL332TOYDUECV2E4MREDHUDR7RIU77Q, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.329596413 (1h34m5.110529789s) [noise=true , address=104.36.82.12:4151 , piece_id=E36QICHR2JJNG5JQUTTOROO75OPUWW4NZL7GP2YAYRP6WASFREHA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.329664489 (1h34m5.110597876s) [noise=true , address=104.36.82.6:4875 , piece_id=IJAEG47NTFFKWVCDUPZTAMNV2Z72IQ74RILA6M7WBH2TMUULDS7Q, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.329728118 (1h34m5.110661496s) [noise=true , address=104.36.82.14:4221 , piece_id=5BRUO5QD2LWA35VTTMH66GIBPU7URG3MGCTBPT4ABYHWWY5RXQMA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.330102417 (1h34m5.111035797s) [noise=true , address=104.36.82.12:4150 , piece_id=QVU6RQZGQV2AONPZ7YXARX3IO3USMVQ2SDWLU2HDZ4D453TU5KHA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.330187936 (1h34m5.111121311s) [noise=true , address=104.36.82.5:4837 , piece_id=XDDXET5PRVZTX7MVP2HGMH6ET47TCT4RJYDLPLFCERRORJC53TXA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:45624->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.330256677 (1h34m5.111190056s) [noise=true , address=77.90.31.16:4733 , piece_id=267J6XV6OMLLP74BJDOTFFSOLRJMVCOWJ3VT5GRBBNALRYONNLUQ, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.330320305 (1h34m5.111253687s) [noise=true , address=77.90.31.11:4554 , piece_id=INZXTKGEQIMD5WDJW4T2GWARHLPUVSQ3KF5OLIMZNAND4R7HL6NA, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.330398162 (1h34m5.111331538s) [noise=true , address=104.36.82.8:4972 , piece_id=C6SIOGACA4B3RXHWFVB6HEHDMELMIAEZT4RIOQ5DDJHDGSGCDE3A, seg_pos={0 0}, upload=1440]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:55370->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:49.330460704 (1h34m5.111394082s) [seg_pos={0 0}, upload=1440]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:25:49.364623532 (1h34m5.145556907s) [upload=1440]: Done uploading segments... Jul 26 23:25:49.364654495 (1h34m5.145588023s) [upload=1440]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.040267107s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....................................XXX.X....XX.X.X.X.......] duration: 29.444944431s FAIL (2025-07-26 23:25:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.445032558s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXX] duration: 1m0.030044707s FAIL (2025-07-26 23:25:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=1428]: Starting upload Jul 26 23:24:28.891700669 (1h32m44.672634047s) [upload=1428]: Done starting upload Jul 26 23:24:28.891967686 (1h32m44.672901054s) [upload=1428]: Uploading segments... Jul 26 23:24:28.892177561 (1h32m44.673110945s) [upload=1428]: Got next segment. Inline: false Jul 26 23:24:28.892230643 (1h32m44.673164008s) [upload=1428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:24:28.892236934 (1h32m44.673170316s) [upload=1428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:24:28.967911395 (1h32m44.748844773s) [seg_pos={0 0}, upload=1428]: Begin upload segment... Jul 26 23:24:28.967987534 (1h32m44.74892091s) [seg_pos={0 0}, upload=1428]: Done begin upload segment. Jul 26 23:24:28.968055462 (1h32m44.748988854s) [noise=true , address=79.127.205.246:4260 , piece_id=W2AIRO6CSQUYIDSUKHYU4NTBJ2TBEUUBJ5IQS7GYBFC7TFX6CLHA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.968071395 (1h32m44.749004773s) [noise=true , address=45.91.116.11:4151 , piece_id=O7K7P6D3TSFA3LMRET2IPN6GNKQB7N7X5UIFDFAITJS4DANHOOXA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.968619727 (1h32m44.749553092s) [noise=true , address=79.127.220.101:4048 , piece_id=CBORT5WYPLVSYWG724TUT5FREX35LZ7ODTL7UQ2R6AVMPAH7J6EQ, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.968644489 (1h32m44.749577886s) [noise=true , address=79.127.205.235:4666 , piece_id=WPPTXGVXVFDWPSTTOMVYC3WNNF6AECHFYLOFVGBNJYNYO63MLB5Q, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.969104081 (1h32m44.750037463s) [noise=true , address=45.91.116.5:4310 , piece_id=EWJVT5X4RXH5M7PKAEDD56LP52DI2OWBGZX2Q6LKBJB7UHVAA45A, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.969205058 (1h32m44.750138444s) [noise=true , address=77.90.31.10:4545 , piece_id=6FYHJVN3ULVLQCPAJVN5TY57S3VXTZVNQSXC245SEG2N5HAD5ZQQ, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.969622051 (1h32m44.750555432s) [noise=true , address=104.36.82.46:4515 , piece_id=6LW3BZ7M3SXFJESMV5TCMIIHFKHQO76ZUKVVQ4LCFCNOJYPJKGSA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.969739530 (1h32m44.750672918s) [noise=true , address=77.90.31.12:4621 , piece_id=AFYMASYMODDQ4O2LH4APCKDRFUUZD7GEJLLK4H6FCTBDKYFFSATQ, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.970000486 (1h32m44.750933868s) [noise=true , address=104.36.82.47:4522 , piece_id=M7D26Q5TX5Q7ZUBFLGR77SD6C5PNQR4MSVGV3JY4GYS6GTDEK2ZA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.970228505 (1h32m44.751161884s) [noise=true , address=104.36.82.54:4175 , piece_id=WVZHYYDZKTM7D4QIE2MDMITBVOAVXIE7FLFCSPLD6H4TQVHAJCIA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.970378986 (1h32m44.751312351s) [noise=true , address=64.49.13.138:4482 , piece_id=5LDM54MDKRLYURBGNEXHU46J2NPTFGP2RYPVZYCTBNUJNEUZIEXQ, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.970609488 (1h32m44.751542864s) [noise=true , address=79.127.219.38:4007 , piece_id=LAKTQFFLNRCOLW7JJF54FSESN5GSJCOYN7IGHVHZCYGHIAYLKOZQ, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.970981704 (1h32m44.751915082s) [noise=true , address=77.90.31.13:4643 , piece_id=7LHUNNTK74SAZC7ZM6AQPDN7WUDLO7C3NBPEL4ATJUW4X3MYSMXA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.971467475 (1h32m44.752400864s) [noise=true , address=79.127.201.215:4357 , piece_id=DPZQ4M5APYS4XC3TZ7R3AD3ZDKZN3KM3ZTLYNVCVKCKW56ZST4RQ, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.972393396 (1h32m44.753326781s) [noise=true , address=156.146.43.225:4448 , piece_id=6WPGJYLUFC6Y3HILYWULJGLLNLJUKJLLJVE4BTPO3URBJHFBVGHA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.975309717 (1h32m44.756243086s) [noise=true , address=79.127.205.245:4241 , piece_id=J7AURBGMYOQIN7RZLRW7K7IKTWBVNMKPGEIV2EZT2WAQ37EJE4AA, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.981936461 (1h32m44.762869846s) [noise=true , address=45.91.116.14:4194 , piece_id=AUSGP6ZX3RUFPHNEHIUBZRW6ZVZES73PWPXGDHWJBTGHJ6UBBJ3Q, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:28.986496651 (1h32m44.767430038s) [noise=true , address=104.36.82.12:4179 , piece_id=GGKBBZ3POQZZWU6JIKVZAKPGVTBQITX5IVY745MBTGW2BKWVXVCQ, seg_pos={0 0}, upload=1428]: Uploading piece... Jul 26 23:24:29.071162362 (1h32m44.852095743s) [upload=1428]: Next returned nil segment Jul 26 23:24:29.071188270 (1h32m44.852121657s) [upload=1428]: Waiting for error group managing segments... Jul 26 23:24:29.089705319 (1h32m44.8706387s) [noise=true , address=79.127.219.38:4007 , piece_id=LAKTQFFLNRCOLW7JJF54FSESN5GSJCOYN7IGHVHZCYGHIAYLKOZQ, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:29.582914978 (1h32m45.363848355s) [noise=true , address=79.127.205.246:4260 , piece_id=W2AIRO6CSQUYIDSUKHYU4NTBJ2TBEUUBJ5IQS7GYBFC7TFX6CLHA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:29.617581067 (1h32m45.398514433s) [noise=true , address=79.127.205.245:4241 , piece_id=J7AURBGMYOQIN7RZLRW7K7IKTWBVNMKPGEIV2EZT2WAQ37EJE4AA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:29.789539565 (1h32m45.570472937s) [noise=true , address=79.127.205.235:4666 , piece_id=WPPTXGVXVFDWPSTTOMVYC3WNNF6AECHFYLOFVGBNJYNYO63MLB5Q, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:30.204055445 (1h32m45.98498882s) [noise=true , address=45.91.116.11:4151 , piece_id=O7K7P6D3TSFA3LMRET2IPN6GNKQB7N7X5UIFDFAITJS4DANHOOXA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:30.343822298 (1h32m46.124755677s) [noise=true , address=156.146.43.225:4448 , piece_id=6WPGJYLUFC6Y3HILYWULJGLLNLJUKJLLJVE4BTPO3URBJHFBVGHA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:31.528757476 (1h32m47.309690844s) [noise=true , address=77.90.31.12:4621 , piece_id=AFYMASYMODDQ4O2LH4APCKDRFUUZD7GEJLLK4H6FCTBDKYFFSATQ, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:32.235727364 (1h32m48.016660726s) [noise=true , address=79.127.220.101:4048 , piece_id=CBORT5WYPLVSYWG724TUT5FREX35LZ7ODTL7UQ2R6AVMPAH7J6EQ, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:33.490637228 (1h32m49.271570606s) [noise=true , address=64.49.13.138:4482 , piece_id=5LDM54MDKRLYURBGNEXHU46J2NPTFGP2RYPVZYCTBNUJNEUZIEXQ, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:40.133309178 (1h32m55.914242555s) [noise=true , address=45.91.116.5:4310 , piece_id=EWJVT5X4RXH5M7PKAEDD56LP52DI2OWBGZX2Q6LKBJB7UHVAA45A, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:42.233843211 (1h32m58.014776587s) [noise=true , address=45.91.116.14:4194 , piece_id=AUSGP6ZX3RUFPHNEHIUBZRW6ZVZES73PWPXGDHWJBTGHJ6UBBJ3Q, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:45.624726874 (1h33m1.405660255s) [noise=true , address=77.90.31.13:4643 , piece_id=7LHUNNTK74SAZC7ZM6AQPDN7WUDLO7C3NBPEL4ATJUW4X3MYSMXA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:24:51.475181510 (1h33m7.256114886s) [noise=true , address=79.127.201.215:4357 , piece_id=DPZQ4M5APYS4XC3TZ7R3AD3ZDKZN3KM3ZTLYNVCVKCKW56ZST4RQ, seg_pos={0 0}, upload=1428]: Done uploading piece. err: <nil> Jul 26 23:25:28.853155158 (1h33m44.634088534s) [noise=true , address=104.36.82.54:4175 , piece_id=WVZHYYDZKTM7D4QIE2MDMITBVOAVXIE7FLFCSPLD6H4TQVHAJCIA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:28.853215699 (1h33m44.634149076s) [noise=true , address=104.36.82.46:4515 , piece_id=6LW3BZ7M3SXFJESMV5TCMIIHFKHQO76ZUKVVQ4LCFCNOJYPJKGSA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: rpc: dial tcp 104.36.82.46:4515: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:28.853268347 (1h33m44.634201724s) [noise=true , address=104.36.82.47:4522 , piece_id=M7D26Q5TX5Q7ZUBFLGR77SD6C5PNQR4MSVGV3JY4GYS6GTDEK2ZA, seg_pos={0 0}, upload=1428]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:28.853326840 (1h33m44.634260215s) [noise=true , address=77.90.31.10:4545 , piece_id=6FYHJVN3ULVLQCPAJVN5TY57S3VXTZVNQSXC245SEG2N5HAD5ZQQ, seg_pos={0 0}, upload=1428]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: rpc: dial tcp 77.90.31.10:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:28.853384120 (1h33m44.634317495s) [noise=true , address=104.36.82.12:4179 , piece_id=GGKBBZ3POQZZWU6JIKVZAKPGVTBQITX5IVY745MBTGW2BKWVXVCQ, seg_pos={0 0}, upload=1428]: Done uploading piece. err: ecclient: upload failed (node:129pP9f3rFfTvP6MhG894N7WHuJRZP8aWhzJf1zBwZ76wbzVbZa, address:104.36.82.12:4179): protocol: write tcp 159.223.143.39:33486->104.36.82.12:4179: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:28.853404741 (1h33m44.63433811s) [seg_pos={0 0}, upload=1428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:25:28.880356894 (1h33m44.661290259s) [upload=1428]: Done uploading segments... Jul 26 23:25:28.880386577 (1h33m44.661319952s) [upload=1428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.030061566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....X........................X......XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.040178158s FAIL (2025-07-26 23:25:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.040212025s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.051839366s FAIL (2025-07-26 23:25:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 434053 (1h33m5.01536743s) [upload=1431]: Uploading segments... Jul 26 23:24:49.234628169 (1h33m5.015561552s) [upload=1431]: Got next segment. Inline: false Jul 26 23:24:49.234717791 (1h33m5.01565117s) [upload=1431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:24:49.234724579 (1h33m5.015657958s) [upload=1431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:24:49.268653350 (1h33m5.049586741s) [seg_pos={0 0}, upload=1431]: Begin upload segment... Jul 26 23:24:49.268707949 (1h33m5.049641325s) [seg_pos={0 0}, upload=1431]: Done begin upload segment. Jul 26 23:24:49.268745762 (1h33m5.049679146s) [noise=true , address=104.36.82.16:4295 , piece_id=CIDXTWSE5CCE662TVGCZYV4KL7DT53TDU44B6DMVY2S73XTIHUWA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.269262640 (1h33m5.050196025s) [noise=true , address=104.36.82.15:4258 , piece_id=ZXT3PXFAUDJUEVXVZYWO2FCUORV4FSNUKRDK3R2HOO5TG5AOQ4NA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.269684220 (1h33m5.050617597s) [noise=true , address=104.36.82.8:4971 , piece_id=ANS2M77NI7BTPM6MX4AQIU4ET3VI5BHT5Q5N2Z5NVOZEJKQMVRVQ, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.270048187 (1h33m5.050981571s) [noise=true , address=77.90.31.14:4663 , piece_id=ACLCPJVL7GPM444MQPQMPDG3DQTPM3X3D43NRR67IYNXCSIOFLLA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.270434940 (1h33m5.051368318s) [noise=true , address=45.91.116.14:4190 , piece_id=XDDY7SH6OENGPESHBYUTWCTGDR3XR5QBHQYO4XSVFLXBUEGDIPZA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.270804948 (1h33m5.05173833s) [noise=true , address=45.91.116.5:4299 , piece_id=ZBD3CNUHNQKCG5U7FRXP6BNBMFBERKEPGGP3GVM3AT54ZZAHYHUQ, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.271244014 (1h33m5.052177398s) [noise=true , address=45.91.116.9:4261 , piece_id=4N3YXEU4VB3YLKNAXS32PRY2UV3FCVZDS7W6KZOUSYZMQB5GKSVA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.271680109 (1h33m5.052613495s) [noise=true , address=104.36.82.10:4078 , piece_id=IEGEG5IHK5P76G5TCCZYRIHH3NB5U233HKN3IQLPFOQ4LVF7AWMQ, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.272080215 (1h33m5.053013605s) [noise=true , address=104.36.82.6:4876 , piece_id=Q6WK7TZ3ZO6VUO747TXHDRASNKS7UVLV233UBZNY776TAOSEXVSA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.272487877 (1h33m5.053421269s) [noise=true , address=45.91.116.12:4227 , piece_id=ZUHM35PQD7I2T2MI2RD5ZZBMZ5HHTD6IRI5BIVR7XWY7ZJJ3L37Q, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.272871658 (1h33m5.053805045s) [noise=true , address=45.91.116.8:4045 , piece_id=WKERGAU6J53WMHNZ2DHD3Q6JUORD36IV7Q7V4L4I354XBNI7TEVA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.273256593 (1h33m5.054189976s) [noise=true , address=45.91.116.10:4083 , piece_id=IVUZRPMQG2JWPX566Y6JS7HPZW6QDGRFNEJYG3Y2QHKWS4F7KAVQ, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.273693285 (1h33m5.054626663s) [noise=true , address=104.36.82.5:4837 , piece_id=32AF62MH5GYBYFG5SF6JYWAN3GLZIVKWNRSJ42FFAST2ATZWZKBQ, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.274091210 (1h33m5.055024595s) [noise=true , address=104.36.82.4:4803 , piece_id=3RBOAD5RPVLEYR7TPEWCJCVR5VJ6SZPQQYHHNQP3FQAWKZTGU4IQ, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.274460638 (1h33m5.055394027s) [noise=true , address=77.90.31.16:4734 , piece_id=GDLDDYZB7XFO2U5YLXZI3SM4TXR65ZDAWA6272ABQITC3IQTKFJA, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.274898862 (1h33m5.055832245s) [noise=true , address=104.36.82.10:4077 , piece_id=HHSOMNB7LFVFIQXPASNAIWPV6JCU6LULX3T5MEAURW546EPT3NQQ, seg_pos={0 0}, upload=1431]: Uploading piece... Jul 26 23:24:49.370758544 (1h33m5.151691909s) [upload=1431]: Next returned nil segment Jul 26 23:24:49.370787398 (1h33m5.151720773s) [upload=1431]: Waiting for error group managing segments... Jul 26 23:24:49.374979196 (1h33m5.155912558s) [noise=true , address=45.91.116.12:4227 , piece_id=ZUHM35PQD7I2T2MI2RD5ZZBMZ5HHTD6IRI5BIVR7XWY7ZJJ3L37Q, seg_pos={0 0}, upload=1431]: Done uploading piece. err: <nil> Jul 26 23:24:49.375505452 (1h33m5.15643882s) [noise=true , address=45.91.116.8:4045 , piece_id=WKERGAU6J53WMHNZ2DHD3Q6JUORD36IV7Q7V4L4I354XBNI7TEVA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: <nil> Jul 26 23:24:49.375662750 (1h33m5.156596119s) [noise=true , address=45.91.116.10:4083 , piece_id=IVUZRPMQG2JWPX566Y6JS7HPZW6QDGRFNEJYG3Y2QHKWS4F7KAVQ, seg_pos={0 0}, upload=1431]: Done uploading piece. err: <nil> Jul 26 23:24:49.375947834 (1h33m5.1568812s) [noise=true , address=45.91.116.14:4190 , piece_id=XDDY7SH6OENGPESHBYUTWCTGDR3XR5QBHQYO4XSVFLXBUEGDIPZA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: <nil> Jul 26 23:24:49.378101683 (1h33m5.159035045s) [noise=true , address=77.90.31.14:4663 , piece_id=ACLCPJVL7GPM444MQPQMPDG3DQTPM3X3D43NRR67IYNXCSIOFLLA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: <nil> Jul 26 23:24:49.823166761 (1h33m5.604100123s) [noise=true , address=45.91.116.5:4299 , piece_id=ZBD3CNUHNQKCG5U7FRXP6BNBMFBERKEPGGP3GVM3AT54ZZAHYHUQ, seg_pos={0 0}, upload=1431]: Done uploading piece. err: <nil> Jul 26 23:24:49.829273573 (1h33m5.610206949s) [noise=true , address=45.91.116.9:4261 , piece_id=4N3YXEU4VB3YLKNAXS32PRY2UV3FCVZDS7W6KZOUSYZMQB5GKSVA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: <nil> Jul 26 23:25:19.206369856 (1h33m34.987303221s) [noise=true , address=77.90.31.16:4734 , piece_id=GDLDDYZB7XFO2U5YLXZI3SM4TXR65ZDAWA6272ABQITC3IQTKFJA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206465831 (1h33m34.98739921s) [noise=true , address=104.36.82.10:4077 , piece_id=HHSOMNB7LFVFIQXPASNAIWPV6JCU6LULX3T5MEAURW546EPT3NQQ, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206535499 (1h33m34.987468876s) [noise=true , address=104.36.82.16:4295 , piece_id=CIDXTWSE5CCE662TVGCZYV4KL7DT53TDU44B6DMVY2S73XTIHUWA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:35510->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206601479 (1h33m34.987534842s) [noise=true , address=104.36.82.6:4876 , piece_id=Q6WK7TZ3ZO6VUO747TXHDRASNKS7UVLV233UBZNY776TAOSEXVSA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:48170->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206697924 (1h33m34.987631303s) [noise=true , address=104.36.82.4:4803 , piece_id=3RBOAD5RPVLEYR7TPEWCJCVR5VJ6SZPQQYHHNQP3FQAWKZTGU4IQ, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206713193 (1h33m34.987646555s) [noise=true , address=104.36.82.5:4837 , piece_id=32AF62MH5GYBYFG5SF6JYWAN3GLZIVKWNRSJ42FFAST2ATZWZKBQ, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206760389 (1h33m34.98769377s) [noise=true , address=104.36.82.15:4258 , piece_id=ZXT3PXFAUDJUEVXVZYWO2FCUORV4FSNUKRDK3R2HOO5TG5AOQ4NA, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206817493 (1h33m34.98775086s) [noise=true , address=104.36.82.8:4971 , piece_id=ANS2M77NI7BTPM6MX4AQIU4ET3VI5BHT5Q5N2Z5NVOZEJKQMVRVQ, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:40442->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206841681 (1h33m34.987775043s) [noise=true , address=104.36.82.10:4078 , piece_id=IEGEG5IHK5P76G5TCCZYRIHH3NB5U233HKN3IQLPFOQ4LVF7AWMQ, seg_pos={0 0}, upload=1431]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:49694->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:25:19.206882676 (1h33m34.987816065s) [seg_pos={0 0}, upload=1431]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:25:19.255542017 (1h33m35.036475385s) [upload=1431]: Done uploading segments... Jul 26 23:25:19.255565482 (1h33m35.036498847s) [upload=1431]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.051863385s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....................................XXX.X....XX.X.X.X......] duration: 12.492422036s FAIL (2025-07-26 23:24:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.492459825s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXX] duration: 1m0.025929669s FAIL (2025-07-26 23:24:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== failed to upload enough pieces (needed at least 14 but got 7) Jul 26 23:23:28.761832675 (1h31m44.542766051s) [upload=1413]: Starting upload Jul 26 23:23:28.761922244 (1h31m44.54285562s) [upload=1413]: Done starting upload Jul 26 23:23:28.762028510 (1h31m44.542961885s) [upload=1413]: Uploading segments... Jul 26 23:23:28.762303916 (1h31m44.543237284s) [upload=1413]: Got next segment. Inline: false Jul 26 23:23:28.762354224 (1h31m44.543287603s) [upload=1413]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:23:28.762361101 (1h31m44.543294477s) [upload=1413]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:23:28.871722446 (1h31m44.652655834s) [seg_pos={0 0}, upload=1413]: Begin upload segment... Jul 26 23:23:28.871786226 (1h31m44.652719603s) [seg_pos={0 0}, upload=1413]: Done begin upload segment. Jul 26 23:23:28.871820238 (1h31m44.652753623s) [noise=true , address=45.91.116.9:4277 , piece_id=YNMVAXXIMC4XHQU6554WEPGTZNTN5TJ2VRXSVXTFFWRGVFN6ND7A, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.871842865 (1h31m44.652776251s) [noise=true , address=77.90.31.13:4629 , piece_id=VNQCQWYOVZZMGOWQVMYTKD3KGYS4UN6QNB5XZDJN5OOEGA3FVJLQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.872472537 (1h31m44.653405924s) [noise=true , address=79.127.205.235:4680 , piece_id=2OU7CCZ63VXBMWIBXZACIKAXMKJBDCR2CVP4EVFQ4VPVMQU6SFFQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.872659972 (1h31m44.653593439s) [noise=true , address=79.127.205.233:4642 , piece_id=RIB45BSKGU4JJEKVEWVJWW3TJ53PXW6LGAEZAA5SW2NQWLELVV6Q, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.872916480 (1h31m44.653849861s) [noise=true , address=104.36.82.52:4117 , piece_id=VVATSMJN365BAGYNIFWQ4FVVESWXL2OTJPR72L75VVAYTWHA3K3A, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.873283836 (1h31m44.654217226s) [noise=true , address=79.127.205.245:4223 , piece_id=TSYE7EY6GV62FSUUP4TQJCVE7C3MJ5X5UQYVKYR2K3VCJS2JKFLQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.873373625 (1h31m44.65430701s) [noise=true , address=79.127.201.214:4321 , piece_id=5N7WXENRQZ4BGG3MUORVMXVNCKYY34K3A6SACHDLPJOFFL5YICQQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.873628324 (1h31m44.65456171s) [noise=true , address=79.127.205.246:4287 , piece_id=3LOIM3VPVOZHIR3ZXADC574LKBPQNEOYRZODKXNPG6OCTWXW7FVA, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.873966664 (1h31m44.654900051s) [noise=true , address=77.90.31.21:4904 , piece_id=RHSVBW4B4ZISGGL4VR6OGJR6P7QN2AO7UYABRXLB5DG7OXRMNTFA, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.874018693 (1h31m44.654952078s) [noise=true , address=77.90.31.18:4826 , piece_id=AHGR7ACF4CHDGZOB6TKXHZRNVE7LUTZENHN7QBAAQUIJ3GF25ORA, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.874387370 (1h31m44.655320756s) [noise=true , address=77.90.31.12:4597 , piece_id=W3PSA6DBLQ4JLSMRKHAKUO5CAUKBI6KKUCLJ4B65SSRKXKZ22UMQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.874542537 (1h31m44.655475926s) [noise=true , address=45.91.116.14:4206 , piece_id=XTC3LLUT2QJ36L6T63NU2M62KRKSEP66FOXY6Z3ZZ3BVHBPGI6BA, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.874765443 (1h31m44.655698829s) [noise=true , address=64.49.13.154:4927 , piece_id=M6HMF2SGFBIVUF63WKCTECD3ILOQC53F7O5U5ESBZ3SLFWK33IMQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.875166565 (1h31m44.656099947s) [noise=true , address=77.90.31.19:4856 , piece_id=O7ZM76KH4OAHR37HU3AEEJJXNRZ2X5JHJTLQL2SPGSULWPRJRGKQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.875198782 (1h31m44.656132145s) [noise=true , address=79.127.201.215:4350 , piece_id=XCTLTIQBP7CAI5GMNSBQQD2S3B6F2EISEXYUZGW3IMYWKSSCVOYQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.875757994 (1h31m44.656691379s) [noise=true , address=156.146.43.225:4448 , piece_id=UY4TXFHLV4PZEPANLQCWKZMKGHOIHHDEXFBP2SVGILUEX3H6VDVQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.876308855 (1h31m44.65724224s) [noise=true , address=104.36.82.47:4518 , piece_id=KLKTZHSCO5GQZ2MIVV6PKEBRPQWLBNJK4XAYAHXMBN5L3GUPMTCQ, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.876847562 (1h31m44.657780946s) [noise=true , address=104.36.82.56:4231 , piece_id=B4REDIHZPYRM55TPOPN3DJY5KPFSM452JLDAZLXJXXLHRZM324OA, seg_pos={0 0}, upload=1413]: Uploading piece... Jul 26 23:23:28.964785095 (1h31m44.745718473s) [upload=1413]: Next returned nil segment Jul 26 23:23:28.964803951 (1h31m44.745737318s) [upload=1413]: Waiting for error group managing segments... Jul 26 23:23:28.981873353 (1h31m44.762806733s) [noise=true , address=77.90.31.18:4826 , piece_id=AHGR7ACF4CHDGZOB6TKXHZRNVE7LUTZENHN7QBAAQUIJ3GF25ORA, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:29.479406273 (1h31m45.260339658s) [noise=true , address=79.127.205.233:4642 , piece_id=RIB45BSKGU4JJEKVEWVJWW3TJ53PXW6LGAEZAA5SW2NQWLELVV6Q, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:29.513081286 (1h31m45.294014666s) [noise=true , address=79.127.205.235:4680 , piece_id=2OU7CCZ63VXBMWIBXZACIKAXMKJBDCR2CVP4EVFQ4VPVMQU6SFFQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:29.516788330 (1h31m45.297721695s) [noise=true , address=79.127.205.245:4223 , piece_id=TSYE7EY6GV62FSUUP4TQJCVE7C3MJ5X5UQYVKYR2K3VCJS2JKFLQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:29.561510213 (1h31m45.342443595s) [noise=true , address=79.127.205.246:4287 , piece_id=3LOIM3VPVOZHIR3ZXADC574LKBPQNEOYRZODKXNPG6OCTWXW7FVA, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:29.856807901 (1h31m45.637741281s) [noise=true , address=64.49.13.154:4927 , piece_id=M6HMF2SGFBIVUF63WKCTECD3ILOQC53F7O5U5ESBZ3SLFWK33IMQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:30.624797509 (1h31m46.405730872s) [noise=true , address=79.127.201.214:4321 , piece_id=5N7WXENRQZ4BGG3MUORVMXVNCKYY34K3A6SACHDLPJOFFL5YICQQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:34.553121649 (1h31m50.334055025s) [noise=true , address=156.146.43.225:4448 , piece_id=UY4TXFHLV4PZEPANLQCWKZMKGHOIHHDEXFBP2SVGILUEX3H6VDVQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:36.814517783 (1h31m52.595451151s) [noise=true , address=45.91.116.9:4277 , piece_id=YNMVAXXIMC4XHQU6554WEPGTZNTN5TJ2VRXSVXTFFWRGVFN6ND7A, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:39.456774908 (1h31m55.237708273s) [noise=true , address=77.90.31.13:4629 , piece_id=VNQCQWYOVZZMGOWQVMYTKD3KGYS4UN6QNB5XZDJN5OOEGA3FVJLQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:46.431623168 (1h32m2.212556533s) [noise=true , address=45.91.116.14:4206 , piece_id=XTC3LLUT2QJ36L6T63NU2M62KRKSEP66FOXY6Z3ZZ3BVHBPGI6BA, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:49.501658387 (1h32m5.282591766s) [noise=true , address=79.127.201.215:4350 , piece_id=XCTLTIQBP7CAI5GMNSBQQD2S3B6F2EISEXYUZGW3IMYWKSSCVOYQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:23:57.454960108 (1h32m13.235893472s) [noise=true , address=77.90.31.19:4856 , piece_id=O7ZM76KH4OAHR37HU3AEEJJXNRZ2X5JHJTLQL2SPGSULWPRJRGKQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: <nil> Jul 26 23:24:28.730215932 (1h32m44.511149312s) [noise=true , address=77.90.31.21:4904 , piece_id=RHSVBW4B4ZISGGL4VR6OGJR6P7QN2AO7UYABRXLB5DG7OXRMNTFA, seg_pos={0 0}, upload=1413]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:28.730320338 (1h32m44.511253729s) [noise=true , address=77.90.31.12:4597 , piece_id=W3PSA6DBLQ4JLSMRKHAKUO5CAUKBI6KKUCLJ4B65SSRKXKZ22UMQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: ecclient: upload failed (node:127QMaS39fDFZixRP6gVMQkZMefSxLo3ToPctZuDmaARY6nB3H9, address:77.90.31.12:4597): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:24:28.730655725 (1h32m44.511589106s) [noise=true , address=104.36.82.52:4117 , piece_id=VVATSMJN365BAGYNIFWQ4FVVESWXL2OTJPR72L75VVAYTWHA3K3A, seg_pos={0 0}, upload=1413]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:104.36.82.52:4117): protocol: write tcp 159.223.143.39:41738->104.36.82.52:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:28.730699044 (1h32m44.511632495s) [noise=true , address=104.36.82.56:4231 , piece_id=B4REDIHZPYRM55TPOPN3DJY5KPFSM452JLDAZLXJXXLHRZM324OA, seg_pos={0 0}, upload=1413]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:28.730730348 (1h32m44.51166373s) [noise=true , address=104.36.82.47:4518 , piece_id=KLKTZHSCO5GQZ2MIVV6PKEBRPQWLBNJK4XAYAHXMBN5L3GUPMTCQ, seg_pos={0 0}, upload=1413]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: rpc: dial tcp 104.36.82.47:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:24:28.730781439 (1h32m44.511714823s) [seg_pos={0 0}, upload=1413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:24:28.753951476 (1h32m44.534884853s) [upload=1413]: Done uploading segments... Jul 26 23:24:28.753976269 (1h32m44.534909644s) [upload=1413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.025960908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.............X.X..........XX...X...X..XX.X...X..X.......X...] duration: 26.041911075s FAIL (2025-07-26 23:24:21 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.041930221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................................XXX.X....XX.X.X.X.....] duration: 24.429335032s FAIL (2025-07-26 23:24:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.429460551s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.02942948s FAIL (2025-07-26 23:24:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== d=1405]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:52120->104.36.82.4:4804: use of closed network connection; context deadline exceeded Jul 26 23:23:24.013462131 (1h31m39.794395513s) [seg_pos={0 0}, upload=1405]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:23:24.041746904 (1h31m39.822680279s) [upload=1405]: Done uploading segments... Jul 26 23:23:24.041776436 (1h31m39.822709801s) [upload=1405]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 26 23:23:46.503524598 (1h32m2.284457976s) [upload=1417]: Starting upload Jul 26 23:23:46.503599927 (1h32m2.284533309s) [upload=1417]: Done starting upload Jul 26 23:23:46.503935402 (1h32m2.284868781s) [upload=1417]: Uploading segments... Jul 26 23:23:46.504083065 (1h32m2.285016462s) [upload=1417]: Got next segment. Inline: false Jul 26 23:23:46.504149495 (1h32m2.28508288s) [upload=1417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:23:46.504157045 (1h32m2.285090435s) [upload=1417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:23:46.534019032 (1h32m2.314952414s) [seg_pos={0 0}, upload=1417]: Begin upload segment... Jul 26 23:23:46.534100410 (1h32m2.315033778s) [seg_pos={0 0}, upload=1417]: Done begin upload segment. Jul 26 23:23:46.534158621 (1h32m2.315092015s) [noise=true , address=104.36.82.11:4114 , piece_id=J5NTNLMURUZ6DKT56BO6GVAJR3WYRVFCQCOD4EC4XP4QYWVBMXPQ, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.534809103 (1h32m2.315742483s) [noise=true , address=77.90.31.10:4517 , piece_id=6SZXB5N7YH7VN6L4XYEL7H6MAHUS3L4AU7FX6LUBUPISMMFH7P2Q, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.534921498 (1h32m2.315854888s) [noise=true , address=45.91.116.10:4083 , piece_id=SIYCXRZ4WLYCAQ4BMK2OZYGP7H5L3C4U3TNFVRGJUIPL62JHJIWQ, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.535331397 (1h32m2.316264792s) [noise=true , address=104.36.82.7:4909 , piece_id=5Y42OLKZHFAIJRDSW322HBKVPZJQISNJXDDCEAPKEHXX7MPX75MQ, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.535446252 (1h32m2.316379637s) [noise=true , address=45.91.116.7:4010 , piece_id=SGROQJ34GWKUIFCSB4JVOHPCFKOZUBMOV27FRG7UIA7HXOF2ZZYA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.535950472 (1h32m2.316883859s) [noise=true , address=104.36.82.7:4912 , piece_id=WSE3W4354Q5BIKH4FQ55SCAJHZISSW67MJADBCKPR7QZJ3FJ5HRA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.536162697 (1h32m2.317096106s) [noise=true , address=77.90.31.15:4698 , piece_id=NN2QSVV4RR3FYBTCAJXES6LJF3MMX7H3QT2JL3A2OXX2URTEJFSA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.536501440 (1h32m2.317434831s) [noise=true , address=104.36.82.9:4006 , piece_id=MLOCDNV3Q3G5FR4XCXCK7CUZQGK5PFM3P7XFLE7FROVECFZM4YVA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.536722422 (1h32m2.317655804s) [noise=true , address=77.90.31.14:4662 , piece_id=H7IXUZRJDAYK7QOERU77HZQAUBGTRS2IYSENZ3MHZSAUSW5CXKQA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.536951675 (1h32m2.317885061s) [noise=true , address=104.36.82.3:4765 , piece_id=SGZ74YFVGRXWOMJFYS4HBKO3UYQNZHK7MQJTSDGWXJAQ7AZK6TDA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.537263491 (1h32m2.318196875s) [noise=true , address=104.36.82.4:4804 , piece_id=TYDV4T5C2FH3Q3T5DWHHBNFLPMBNDPP2WX77WN2VS56QB7HYJHRA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.537401539 (1h32m2.318334928s) [noise=true , address=45.91.116.4:4939 , piece_id=HUWF5GBPNLH3W46VSNCLUNRVXDLQR33ZLQK5N7RNGUSQYE736TBA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.537784994 (1h32m2.318718374s) [noise=true , address=45.91.116.11:4118 , piece_id=TO2RHBOCSSNV2QHFBTJJDOV6QYIT3U6THAPET4ROB2G7CZFW3NJQ, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.537812980 (1h32m2.318746366s) [noise=true , address=104.36.82.3:4767 , piece_id=MU3FKXVVOOASGT2PMXW2CRFBQHRL3SYXRIUKWJXRMTRLXOIZRX7A, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.538207446 (1h32m2.319140834s) [noise=true , address=45.91.116.8:4045 , piece_id=FFALOSOXZGYMM5CQT5GBXSCFFHJ2LDLZUWBHEQV36SMVI75D4KBA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.538589505 (1h32m2.31952289s) [noise=true , address=45.91.116.6:4975 , piece_id=IAUJTG3N7WS4BUXNGNODB433QK66LIU32I3HNMG25K5ISAXMFODA, seg_pos={0 0}, upload=1417]: Uploading piece... Jul 26 23:23:46.672527687 (1h32m2.453461072s) [upload=1417]: Next returned nil segment Jul 26 23:23:46.672577452 (1h32m2.453510838s) [upload=1417]: Waiting for error group managing segments... Jul 26 23:23:46.677014989 (1h32m2.457948369s) [noise=true , address=45.91.116.4:4939 , piece_id=HUWF5GBPNLH3W46VSNCLUNRVXDLQR33ZLQK5N7RNGUSQYE736TBA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:23:46.677750779 (1h32m2.458684158s) [noise=true , address=45.91.116.8:4045 , piece_id=FFALOSOXZGYMM5CQT5GBXSCFFHJ2LDLZUWBHEQV36SMVI75D4KBA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:23:46.678282219 (1h32m2.459215589s) [noise=true , address=45.91.116.6:4975 , piece_id=IAUJTG3N7WS4BUXNGNODB433QK66LIU32I3HNMG25K5ISAXMFODA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:23:46.678778246 (1h32m2.459711608s) [noise=true , address=45.91.116.7:4010 , piece_id=SGROQJ34GWKUIFCSB4JVOHPCFKOZUBMOV27FRG7UIA7HXOF2ZZYA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:23:46.679158095 (1h32m2.460091472s) [noise=true , address=45.91.116.11:4118 , piece_id=TO2RHBOCSSNV2QHFBTJJDOV6QYIT3U6THAPET4ROB2G7CZFW3NJQ, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:23:46.679818095 (1h32m2.460751479s) [noise=true , address=77.90.31.14:4662 , piece_id=H7IXUZRJDAYK7QOERU77HZQAUBGTRS2IYSENZ3MHZSAUSW5CXKQA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:23:46.680443617 (1h32m2.461376983s) [noise=true , address=45.91.116.10:4083 , piece_id=SIYCXRZ4WLYCAQ4BMK2OZYGP7H5L3C4U3TNFVRGJUIPL62JHJIWQ, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:23:46.725067367 (1h32m2.506000744s) [noise=true , address=77.90.31.15:4698 , piece_id=NN2QSVV4RR3FYBTCAJXES6LJF3MMX7H3QT2JL3A2OXX2URTEJFSA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: <nil> Jul 26 23:24:16.468391765 (1h32m32.249325149s) [noise=true , address=104.36.82.4:4804 , piece_id=TYDV4T5C2FH3Q3T5DWHHBNFLPMBNDPP2WX77WN2VS56QB7HYJHRA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468474255 (1h32m32.249407636s) [noise=true , address=104.36.82.3:4767 , piece_id=MU3FKXVVOOASGT2PMXW2CRFBQHRL3SYXRIUKWJXRMTRLXOIZRX7A, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468547111 (1h32m32.249480478s) [noise=true , address=77.90.31.10:4517 , piece_id=6SZXB5N7YH7VN6L4XYEL7H6MAHUS3L4AU7FX6LUBUPISMMFH7P2Q, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468613451 (1h32m32.249546842s) [noise=true , address=104.36.82.3:4765 , piece_id=SGZ74YFVGRXWOMJFYS4HBKO3UYQNZHK7MQJTSDGWXJAQ7AZK6TDA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468721923 (1h32m32.249655307s) [noise=true , address=104.36.82.11:4114 , piece_id=J5NTNLMURUZ6DKT56BO6GVAJR3WYRVFCQCOD4EC4XP4QYWVBMXPQ, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468798457 (1h32m32.249731835s) [noise=true , address=104.36.82.7:4912 , piece_id=WSE3W4354Q5BIKH4FQ55SCAJHZISSW67MJADBCKPR7QZJ3FJ5HRA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468813705 (1h32m32.249747093s) [noise=true , address=104.36.82.7:4909 , piece_id=5Y42OLKZHFAIJRDSW322HBKVPZJQISNJXDDCEAPKEHXX7MPX75MQ, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468907899 (1h32m32.249841277s) [noise=true , address=104.36.82.9:4006 , piece_id=MLOCDNV3Q3G5FR4XCXCK7CUZQGK5PFM3P7XFLE7FROVECFZM4YVA, seg_pos={0 0}, upload=1417]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:24:16.468940458 (1h32m32.249873847s) [seg_pos={0 0}, upload=1417]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:24:16.494948736 (1h32m32.275882119s) [upload=1417]: Done uploading segments... Jul 26 23:24:16.494975722 (1h32m32.275909106s) [upload=1417]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.029458148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 24 out of 60 is failed (threshold=3) [......X........................X......XXXXXXXXXXXXXXXXXXXXXX] duration: 30.027099728s FAIL (2025-07-26 23:24:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.027148508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.033335448s FAIL (2025-07-26 23:23:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== in upload segment. Jul 26 23:22:54.076739636 (1h31m9.857673022s) [noise=true , address=45.91.116.5:4298 , piece_id=HLTSKSO2ANY55B6PIXMFU237RSB2GOG44ZXKP7J6XUCBPD23X4VA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.077064856 (1h31m9.857998225s) [noise=true , address=104.36.82.6:4874 , piece_id=5F7EB5FG6DCL5U52JW35TIDXGGIQIVZFXUDUMTESW3SUETBVRSNQ, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.077586130 (1h31m9.858519492s) [noise=true , address=77.90.31.16:4734 , piece_id=QVTS7CWR74AFEMY6MDWIT7ZL4VJVUFZXHW5FU7OXKIFHKFAL5INQ, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.079118640 (1h31m9.860052017s) [noise=true , address=45.91.116.7:4011 , piece_id=FELJPQ2ORBPZJMXWBVYJ4QFSA6NUUBL3TSS56EZ65IY7RSD6IZEQ, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.079817042 (1h31m9.860750404s) [noise=true , address=104.36.82.2:4732 , piece_id=ONR5I6MRPZZHNZF4PE4MVTGWRD7DYR2NSGUTKRDOIUYWQXJRI64A, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.080604529 (1h31m9.861537891s) [noise=true , address=104.36.82.14:4223 , piece_id=JM5II7NWMTC6ZT4CC55VEGX4O5UUDYJKVBAEAJQCYA7QWEC65R3A, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.081252608 (1h31m9.862185986s) [noise=true , address=104.36.82.3:4765 , piece_id=WXAJEYTG7MWNZY4RNPV6JXEBQM3EO5NOBD7WNMWCY3OYLFLJQNOA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.081917503 (1h31m9.862850875s) [noise=true , address=45.91.116.4:4937 , piece_id=JELGBORAHLPSH7IN5T7HYAE44JQXXXFQ4FFMDQ6GQOZHOJR5ZBUA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.082574461 (1h31m9.863507823s) [noise=true , address=104.36.82.9:4007 , piece_id=FPFXAGCWN6FP32ES2PPG3RYSTSMFKY7XFE4GKDH47NFBPKVQZRXA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.083346443 (1h31m9.864279806s) [noise=true , address=45.91.116.8:4048 , piece_id=DM36EQ7BSLEKKBV7LHRTCWIC33DB3CA27P4VVF732DV5Q5EB4KXA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.084500504 (1h31m9.865433868s) [noise=true , address=104.36.82.5:4839 , piece_id=XFPRB6ZZZW2YAT6LR5B4CYV5KVF4HGXTZ7BSPSPDBVRP3NFY6ZJA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.085670370 (1h31m9.866603742s) [noise=true , address=45.91.116.8:4045 , piece_id=U5XHJKZDLQIX6RQ7LU4DDDEDYL4EVNS5PKQU3E2PMPH2FBB4AUQA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.086174398 (1h31m9.867107768s) [noise=true , address=104.36.82.12:4151 , piece_id=ZPYFGBOI7RBKBDYYX365PVFC5CGABR4NUG7EFRODY7QI7M2SHYYA, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.087003900 (1h31m9.86793727s) [noise=true , address=104.36.82.7:4912 , piece_id=TSG7A2WFT6TKHWMPWKLOPL572ACTNODAO323Q4722FVS4Y3VNIVQ, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.087789225 (1h31m9.868722602s) [noise=true , address=104.36.82.4:4804 , piece_id=Y7Y4YSEOAIFB5HOUY4PE52V5LJRDQOSNJIOYMHNJRHKKR4YLCG2Q, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.088619516 (1h31m9.869552882s) [noise=true , address=104.36.82.13:4186 , piece_id=M7E7TH5FIR7NCCZK5SRTTGHE3OXBFAOQL2V6IB2TOJQX7XLLFPNQ, seg_pos={0 0}, upload=1405]: Uploading piece... Jul 26 23:22:54.159657673 (1h31m9.940591037s) [upload=1405]: Next returned nil segment Jul 26 23:22:54.160515722 (1h31m9.941449112s) [upload=1405]: Waiting for error group managing segments... Jul 26 23:22:54.164170958 (1h31m9.945104319s) [noise=true , address=45.91.116.8:4045 , piece_id=U5XHJKZDLQIX6RQ7LU4DDDEDYL4EVNS5PKQU3E2PMPH2FBB4AUQA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: <nil> Jul 26 23:22:54.164685278 (1h31m9.945618646s) [noise=true , address=45.91.116.4:4937 , piece_id=JELGBORAHLPSH7IN5T7HYAE44JQXXXFQ4FFMDQ6GQOZHOJR5ZBUA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: <nil> Jul 26 23:22:54.165139680 (1h31m9.94607305s) [noise=true , address=45.91.116.7:4011 , piece_id=FELJPQ2ORBPZJMXWBVYJ4QFSA6NUUBL3TSS56EZ65IY7RSD6IZEQ, seg_pos={0 0}, upload=1405]: Done uploading piece. err: <nil> Jul 26 23:22:54.165518292 (1h31m9.946451657s) [noise=true , address=45.91.116.8:4048 , piece_id=DM36EQ7BSLEKKBV7LHRTCWIC33DB3CA27P4VVF732DV5Q5EB4KXA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: <nil> Jul 26 23:22:54.310043183 (1h31m10.090976546s) [noise=true , address=45.91.116.5:4298 , piece_id=HLTSKSO2ANY55B6PIXMFU237RSB2GOG44ZXKP7J6XUCBPD23X4VA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: <nil> Jul 26 23:23:24.012599698 (1h31m39.793533081s) [noise=true , address=104.36.82.5:4839 , piece_id=XFPRB6ZZZW2YAT6LR5B4CYV5KVF4HGXTZ7BSPSPDBVRP3NFY6ZJA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.012681521 (1h31m39.793614897s) [noise=true , address=104.36.82.6:4874 , piece_id=5F7EB5FG6DCL5U52JW35TIDXGGIQIVZFXUDUMTESW3SUETBVRSNQ, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:52358->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.012749421 (1h31m39.7936828s) [noise=true , address=104.36.82.2:4732 , piece_id=ONR5I6MRPZZHNZF4PE4MVTGWRD7DYR2NSGUTKRDOIUYWQXJRI64A, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:40504->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.012808929 (1h31m39.793742294s) [noise=true , address=104.36.82.12:4151 , piece_id=ZPYFGBOI7RBKBDYYX365PVFC5CGABR4NUG7EFRODY7QI7M2SHYYA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.012866774 (1h31m39.793800141s) [noise=true , address=77.90.31.16:4734 , piece_id=QVTS7CWR74AFEMY6MDWIT7ZL4VJVUFZXHW5FU7OXKIFHKFAL5INQ, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 159.223.143.39:58228->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.012925428 (1h31m39.793858796s) [noise=true , address=104.36.82.14:4223 , piece_id=JM5II7NWMTC6ZT4CC55VEGX4O5UUDYJKVBAEAJQCYA7QWEC65R3A, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:35366->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.012985168 (1h31m39.793918545s) [noise=true , address=104.36.82.3:4765 , piece_id=WXAJEYTG7MWNZY4RNPV6JXEBQM3EO5NOBD7WNMWCY3OYLFLJQNOA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:56728->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.013043326 (1h31m39.793976703s) [noise=true , address=104.36.82.9:4007 , piece_id=FPFXAGCWN6FP32ES2PPG3RYSTSMFKY7XFE4GKDH47NFBPKVQZRXA, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:37536->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.013110036 (1h31m39.794043403s) [noise=true , address=104.36.82.7:4912 , piece_id=TSG7A2WFT6TKHWMPWKLOPL572ACTNODAO323Q4722FVS4Y3VNIVQ, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.013167520 (1h31m39.794100885s) [noise=true , address=104.36.82.13:4186 , piece_id=M7E7TH5FIR7NCCZK5SRTTGHE3OXBFAOQL2V6IB2TOJQX7XLLFPNQ, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:38388->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:24.013428505 (1h31m39.794361869s) [noise=true , address=104.36.82.4:4804 , piece_id=Y7Y4YSEOAIFB5HOUY4PE52V5LJRDQOSNJIOYMHNJRHKKR4YLCG2Q, seg_pos={0 0}, upload=1405]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:52120->104.36.82.4:4804: use of closed network connection; context deadline exceeded Jul 26 23:23:24.013462131 (1h31m39.794395513s) [seg_pos={0 0}, upload=1405]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:23:24.041746904 (1h31m39.822680279s) [upload=1405]: Done uploading segments... Jul 26 23:23:24.041776436 (1h31m39.822709801s) [upload=1405]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.033363054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXX] duration: 1m0.029346358s FAIL (2025-07-26 23:23:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== .. Jul 26 23:22:21.549565533 (1h30m37.330498921s) [noise=true , address=104.36.82.52:4094 , piece_id=NIK73VTYJNLJ5UORJYSTP2BQTUNF5NBGFRREASEAPFDRYKKBOIMQ, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.549589437 (1h30m37.330522822s) [noise=true , address=104.36.82.53:4132 , piece_id=XFQDCRGBMAKAQGTZLBHUXDI2O5PVGGUSSVNPMTGPEDKREM3GBLPA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.549910681 (1h30m37.330844066s) [noise=true , address=77.90.31.15:4729 , piece_id=NF2D3NHF7EOIRER5LY5INKCH5RUHDA6JFSTGVA3JJHHTCY7BOFAA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.550114014 (1h30m37.3310474s) [noise=true , address=104.36.82.55:4218 , piece_id=FMPOUJBYUPDEMB7LAYBC45K3XTT5ZT4HJJF4CTNW2FSRVNKYDHPA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.550261714 (1h30m37.331195128s) [noise=true , address=104.36.82.56:4249 , piece_id=X43WURZBBKS5VV2TJRKYGULIPSZQSMPHIVYAGBQ54CDUW3DISUIQ, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.550599449 (1h30m37.331532832s) [noise=true , address=64.49.13.140:4508 , piece_id=UP76BMNB4TJ7EEKIFU5BQZAGC5I5NZ4NRP272QYMRXA5MFPROPOA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.550611244 (1h30m37.331544628s) [noise=true , address=104.36.82.35:4581 , piece_id=WTLUP6UDPV3TLYHTSAT7WBHXATTRMX3QGU7PJK44QIUDUYSK2QEA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.551191867 (1h30m37.332125249s) [noise=true , address=104.36.82.21:4493 , piece_id=T7MNQEZ72JAYAJ6X7NDSVZZDEODFGH5XNRENELJAFQONMO2DOV5A, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.551706103 (1h30m37.332639489s) [noise=true , address=79.127.205.245:4242 , piece_id=AHNFRHL2LDX7DL6SHGEOAUZ3KH464ZEP5TNKBRK2F46SAKIVIMMA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.552311289 (1h30m37.333244674s) [noise=true , address=77.90.31.11:4580 , piece_id=CEUS3HM6O4FXOYZY6PG6IG5KG64CR4RAPXAK4HW52VJOA4Z6OLPQ, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.552866359 (1h30m37.333799732s) [noise=true , address=104.36.82.47:4532 , piece_id=M3MGJ5GQ6I7KYT7RZOO2ZED2PZCY5GKIFKKSXYO2PJ6L4PKHTAIA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.553133057 (1h30m37.334066425s) [noise=true , address=104.36.82.41:4783 , piece_id=344WH6WLT6F5T6YOX2S5Y6JHR5LKANHETXFPNB5UHMGFMZRUYZBA, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.553748520 (1h30m37.334681904s) [noise=true , address=77.90.31.12:4593 , piece_id=MRKZPFH4HNCJ6H2UZKNWOFNGMB46FUMEJAU4NRDTDWRHUXPZ433A, seg_pos={0 0}, upload=1396]: Uploading piece... Jul 26 23:22:21.611947700 (1h30m37.392881077s) [upload=1396]: Next returned nil segment Jul 26 23:22:21.611965271 (1h30m37.392898652s) [upload=1396]: Waiting for error group managing segments... Jul 26 23:22:21.615677072 (1h30m37.396610456s) [noise=true , address=77.90.31.15:4729 , piece_id=NF2D3NHF7EOIRER5LY5INKCH5RUHDA6JFSTGVA3JJHHTCY7BOFAA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: <nil> Jul 26 23:22:21.672469783 (1h30m37.453403145s) [noise=true , address=77.90.31.11:4580 , piece_id=CEUS3HM6O4FXOYZY6PG6IG5KG64CR4RAPXAK4HW52VJOA4Z6OLPQ, seg_pos={0 0}, upload=1396]: Done uploading piece. err: <nil> Jul 26 23:22:21.685802882 (1h30m37.466736251s) [noise=true , address=77.90.31.12:4593 , piece_id=MRKZPFH4HNCJ6H2UZKNWOFNGMB46FUMEJAU4NRDTDWRHUXPZ433A, seg_pos={0 0}, upload=1396]: Done uploading piece. err: <nil> Jul 26 23:22:22.036790351 (1h30m37.817723717s) [noise=true , address=45.91.116.11:4151 , piece_id=KMCVJKXSNBXL5R2SNZJ7M4AHJOGR2M4SGKI5HIOYEPKPPOTGU3GQ, seg_pos={0 0}, upload=1396]: Done uploading piece. err: <nil> Jul 26 23:22:22.112873730 (1h30m37.893807109s) [noise=true , address=79.127.205.246:4281 , piece_id=NSG5ZXSPFAA622GRBINPP7Q75PX3OILDDH6D7JRV6AXYBABL7UQA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: <nil> Jul 26 23:22:22.133037992 (1h30m37.91397162s) [noise=true , address=79.127.205.245:4242 , piece_id=AHNFRHL2LDX7DL6SHGEOAUZ3KH464ZEP5TNKBRK2F46SAKIVIMMA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: <nil> Jul 26 23:22:28.991232920 (1h30m44.772166293s) [noise=true , address=64.49.13.140:4508 , piece_id=UP76BMNB4TJ7EEKIFU5BQZAGC5I5NZ4NRP272QYMRXA5MFPROPOA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: <nil> Jul 26 23:23:21.470781415 (1h31m37.25171479s) [noise=true , address=104.36.82.56:4249 , piece_id=X43WURZBBKS5VV2TJRKYGULIPSZQSMPHIVYAGBQ54CDUW3DISUIQ, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 159.223.143.39:49052->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.470841009 (1h31m37.251774397s) [noise=true , address=104.36.82.47:4532 , piece_id=M3MGJ5GQ6I7KYT7RZOO2ZED2PZCY5GKIFKKSXYO2PJ6L4PKHTAIA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 159.223.143.39:40350->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.470901270 (1h31m37.251834645s) [noise=true , address=104.36.82.35:4581 , piece_id=WTLUP6UDPV3TLYHTSAT7WBHXATTRMX3QGU7PJK44QIUDUYSK2QEA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:104.36.82.35:4581): protocol: write tcp 159.223.143.39:35868->104.36.82.35:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.470914984 (1h31m37.25184837s) [noise=true , address=104.36.82.14:4256 , piece_id=VFYQC5BXDH5PPYEBKSW2STHZTUDBEH2USUNQPU67BQUXLCSY54GA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: write tcp 159.223.143.39:43484->104.36.82.14:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.470958322 (1h31m37.251891687s) [noise=true , address=104.36.82.21:4493 , piece_id=T7MNQEZ72JAYAJ6X7NDSVZZDEODFGH5XNRENELJAFQONMO2DOV5A, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD, address:104.36.82.21:4493): protocol: write tcp 159.223.143.39:51012->104.36.82.21:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.470974297 (1h31m37.251907683s) [noise=true , address=104.36.82.52:4094 , piece_id=NIK73VTYJNLJ5UORJYSTP2BQTUNF5NBGFRREASEAPFDRYKKBOIMQ, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 159.223.143.39:57992->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.471061238 (1h31m37.251994613s) [noise=true , address=104.36.82.54:4192 , piece_id=G6J5PNPORH5LE7SZ2U4TDH2LMO73PDDMSOVKJ76F4T53GKPJWLMA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: write tcp 159.223.143.39:55358->104.36.82.54:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.471115608 (1h31m37.252048986s) [noise=true , address=104.36.82.55:4218 , piece_id=FMPOUJBYUPDEMB7LAYBC45K3XTT5ZT4HJJF4CTNW2FSRVNKYDHPA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 159.223.143.39:54934->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.471126296 (1h31m37.252059675s) [noise=true , address=104.36.82.41:4783 , piece_id=344WH6WLT6F5T6YOX2S5Y6JHR5LKANHETXFPNB5UHMGFMZRUYZBA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: write tcp 159.223.143.39:44484->104.36.82.41:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.471135206 (1h31m37.252068581s) [noise=true , address=104.36.82.46:4501 , piece_id=HISYKVWDBUSPKTDV7PE3GYBJOWQEAOJXMT2LRLQC5XNDU7XTBIHQ, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 159.223.143.39:50058->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.471154125 (1h31m37.252087493s) [noise=true , address=104.36.82.53:4132 , piece_id=XFQDCRGBMAKAQGTZLBHUXDI2O5PVGGUSSVNPMTGPEDKREM3GBLPA, seg_pos={0 0}, upload=1396]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: write tcp 159.223.143.39:46192->104.36.82.53:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:23:21.471185409 (1h31m37.252118785s) [seg_pos={0 0}, upload=1396]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:23:21.497921904 (1h31m37.278855272s) [upload=1396]: Done uploading segments... Jul 26 23:23:21.497946742 (1h31m37.27888011s) [upload=1396]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.029363093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..............X.X..........XX...X...X..XX.X...X..X.......X..] duration: 7.619684698s FAIL (2025-07-26 23:23:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.619707862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......................................XXX.X....XX.X.X.X....] duration: 13.451240455s FAIL (2025-07-26 23:23:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.451300877s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...............X.X..........XX...X...X..XX.X...X..X.......X.] duration: 42.26721025s FAIL (2025-07-26 23:23:10 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.267229414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [.......X........................X......XXXXXXXXXXXXXXXXXXXXX] duration: 30.026495313s FAIL (2025-07-26 23:22:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.026529339s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.026539942s FAIL (2025-07-26 23:22:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 23:21:54.887654984 (1h30m10.66858837s) [upload=1383]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:22:18.565715588 (1h30m34.34664897s) [upload=1395]: Starting upload Jul 26 23:22:18.565801804 (1h30m34.346735183s) [upload=1395]: Done starting upload Jul 26 23:22:18.565871378 (1h30m34.346804758s) [upload=1395]: Uploading segments... Jul 26 23:22:18.566111101 (1h30m34.347044488s) [upload=1395]: Got next segment. Inline: false Jul 26 23:22:18.566169255 (1h30m34.347102637s) [upload=1395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:18.566175426 (1h30m34.347108808s) [upload=1395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:18.676953085 (1h30m34.457886484s) [seg_pos={0 0}, upload=1395]: Begin upload segment... Jul 26 23:22:18.677006845 (1h30m34.457940224s) [seg_pos={0 0}, upload=1395]: Done begin upload segment. Jul 26 23:22:18.677041582 (1h30m34.457974988s) [noise=true , address=104.36.82.2:4729 , piece_id=KRD4T373WUGA2NXAT6OJ6AA52PBZRIIIZIPGN2YFQZOILITCTO2Q, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677062416 (1h30m34.457995802s) [noise=true , address=45.91.116.6:4975 , piece_id=CLF73O5UP4RJU4WAXYWGBY6NC6BAWVIC3JZ4H4HS6FOPEURR3USA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677452298 (1h30m34.458385685s) [noise=true , address=45.91.116.9:4262 , piece_id=NFNHJSCWMYC72IT3ZPIAFJ76TRDS7W5ETEML7VVNAQBWNCMNO5ZQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677540134 (1h30m34.458473515s) [noise=true , address=45.91.116.10:4083 , piece_id=E5MWNTRIFLRIJFI264QMXFHYZF55K64G4T2ZOB4JYSUONSDXYG7Q, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677775760 (1h30m34.458709135s) [noise=true , address=77.90.31.14:4662 , piece_id=DOU77WHHHMO4OZHWUGA7F2BEB42J536JI2IAB4NBMKZ6CZDKDYOA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677883425 (1h30m34.45881681s) [noise=true , address=104.36.82.14:4221 , piece_id=IPQISEAZ2XSCNQDIQNM6GRAOKLCQGZUFB6L75QA2MMIEMZ75DFLQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678076946 (1h30m34.459010321s) [noise=true , address=104.36.82.4:4802 , piece_id=ZUFZAVMBSVYYJQTXIQTACIMPGHGVZUFLETPGMPZEY42GPXC6CI2A, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678267530 (1h30m34.459200911s) [noise=true , address=104.36.82.9:4007 , piece_id=MG2W3UGJANGX4PXHA67KR5RHVUJ5EUZJXZDZKRVXWM6RMADBDNHA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678401502 (1h30m34.459334877s) [noise=true , address=104.36.82.16:4293 , piece_id=EBHNEPXIKLB6MJRXKYH2RRJKDUA4PBCIELYDGS2JORPCLHF2PVHA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678703139 (1h30m34.459636517s) [noise=true , address=104.36.82.2:4732 , piece_id=FNIRCO4URPUS4JYLBAP6V5WMFOI34ZKV4PNVFUM7C5JQ24F674OQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678984081 (1h30m34.459917464s) [noise=true , address=104.36.82.12:4151 , piece_id=SGTWR6MYRNRSNPVLI6E4UGKTKQPSQ7TREG6WYPH53BYHGJPPH7WA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.679373248 (1h30m34.460306624s) [noise=true , address=45.91.116.8:4047 , piece_id=GBCSXHVMJMC7MDGSYKH3A4VZI7UE3VD352HW3TRI64D55JJPOQCA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.679719787 (1h30m34.460653169s) [noise=true , address=45.91.116.4:4939 , piece_id=G63GNEM3DWU2FQD5WEJPVG6LBAVG4US4CDCDU3H3H6PNS3QC2TIA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.680159105 (1h30m34.46109249s) [noise=true , address=104.36.82.6:4874 , piece_id=SD5E3FZBMAPJ3KITOX7BHUD6KGK7XHKOQ6DZQYJ2FBPQCVBSU7GA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.680605455 (1h30m34.46153883s) [noise=true , address=45.91.116.14:4191 , piece_id=KQVN73L42RDTC2RNAXSHBG3IZYDKGVSYV2PSQD6TF6OP27LXHQ3A, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.680942854 (1h30m34.461876236s) [noise=true , address=45.91.116.5:4298 , piece_id=RJNAE3UYHYRO37A675HIGSYWAIVO4ZD4ZNVFIHJ6KUDE27YBL6GQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.749684604 (1h30m34.530617979s) [upload=1395]: Next returned nil segment Jul 26 23:22:18.749717216 (1h30m34.530650592s) [upload=1395]: Waiting for error group managing segments... Jul 26 23:22:18.753292333 (1h30m34.534225702s) [noise=true , address=45.91.116.10:4083 , piece_id=E5MWNTRIFLRIJFI264QMXFHYZF55K64G4T2ZOB4JYSUONSDXYG7Q, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.753340094 (1h30m34.534273464s) [noise=true , address=45.91.116.8:4047 , piece_id=GBCSXHVMJMC7MDGSYKH3A4VZI7UE3VD352HW3TRI64D55JJPOQCA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.753726053 (1h30m34.534659422s) [noise=true , address=45.91.116.4:4939 , piece_id=G63GNEM3DWU2FQD5WEJPVG6LBAVG4US4CDCDU3H3H6PNS3QC2TIA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.753990707 (1h30m34.534924076s) [noise=true , address=45.91.116.6:4975 , piece_id=CLF73O5UP4RJU4WAXYWGBY6NC6BAWVIC3JZ4H4HS6FOPEURR3USA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.755955656 (1h30m34.536889025s) [noise=true , address=77.90.31.14:4662 , piece_id=DOU77WHHHMO4OZHWUGA7F2BEB42J536JI2IAB4NBMKZ6CZDKDYOA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.880443516 (1h30m34.661376884s) [noise=true , address=45.91.116.14:4191 , piece_id=KQVN73L42RDTC2RNAXSHBG3IZYDKGVSYV2PSQD6TF6OP27LXHQ3A, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:19.000935049 (1h30m34.781868418s) [noise=true , address=45.91.116.5:4298 , piece_id=RJNAE3UYHYRO37A675HIGSYWAIVO4ZD4ZNVFIHJ6KUDE27YBL6GQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:19.072966588 (1h30m34.853899964s) [noise=true , address=45.91.116.9:4262 , piece_id=NFNHJSCWMYC72IT3ZPIAFJ76TRDS7W5ETEML7VVNAQBWNCMNO5ZQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:48.533008871 (1h31m4.313942248s) [noise=true , address=104.36.82.14:4221 , piece_id=IPQISEAZ2XSCNQDIQNM6GRAOKLCQGZUFB6L75QA2MMIEMZ75DFLQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533104583 (1h31m4.314037967s) [noise=true , address=104.36.82.2:4732 , piece_id=FNIRCO4URPUS4JYLBAP6V5WMFOI34ZKV4PNVFUM7C5JQ24F674OQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533207069 (1h31m4.31414045s) [noise=true , address=104.36.82.12:4151 , piece_id=SGTWR6MYRNRSNPVLI6E4UGKTKQPSQ7TREG6WYPH53BYHGJPPH7WA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533280800 (1h31m4.314214168s) [noise=true , address=104.36.82.6:4874 , piece_id=SD5E3FZBMAPJ3KITOX7BHUD6KGK7XHKOQ6DZQYJ2FBPQCVBSU7GA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533359742 (1h31m4.314293113s) [noise=true , address=104.36.82.2:4729 , piece_id=KRD4T373WUGA2NXAT6OJ6AA52PBZRIIIZIPGN2YFQZOILITCTO2Q, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533421750 (1h31m4.31435512s) [noise=true , address=104.36.82.4:4802 , piece_id=ZUFZAVMBSVYYJQTXIQTACIMPGHGVZUFLETPGMPZEY42GPXC6CI2A, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533503864 (1h31m4.314437229s) [noise=true , address=104.36.82.16:4293 , piece_id=EBHNEPXIKLB6MJRXKYH2RRJKDUA4PBCIELYDGS2JORPCLHF2PVHA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533565729 (1h31m4.314499092s) [noise=true , address=104.36.82.9:4007 , piece_id=MG2W3UGJANGX4PXHA67KR5RHVUJ5EUZJXZDZKRVXWM6RMADBDNHA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:48376->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533589676 (1h31m4.314523067s) [seg_pos={0 0}, upload=1395]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:22:48.556625151 (1h31m4.337558522s) [upload=1395]: Done uploading segments... Jul 26 23:22:48.556653190 (1h31m4.337586558s) [upload=1395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.026567198s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........................................XXX.X....XX.X.X.X...] duration: 12.649190372s FAIL (2025-07-26 23:22:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.649211487s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [................X.X..........XX...X...X..XX.X...X..X.......X] duration: 1m0.003043777s FAIL (2025-07-26 23:22:28 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.003081415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [........X........................X......XXXXXXXXXXXXXXXXXXXX] duration: 30.04204282s FAIL (2025-07-26 23:22:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.04207014s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXX] duration: 1m0.036548333s FAIL (2025-07-26 23:22:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== VOYIJAF3UBWMAUQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.528190591 (1h29m37.309123971s) [noise=true , address=77.90.31.10:4536 , piece_id=35F4NXUJ3NH7SU27OZAJPU2GWOACHL4AKAX7TNE2PCYZFLXVONHA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.529363038 (1h29m37.310296422s) [noise=true , address=156.146.43.225:4449 , piece_id=DJGEZPD5P64X2WIHWCRXALEHVEGR5NHPGRLMA5AJPDZMOYHSOC6A, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.529914695 (1h29m37.310848077s) [noise=true , address=104.36.82.52:4120 , piece_id=MVPJLSLL65XNXKNEV5PPWGWZZFEX625WSK46RZPGBNX3D6UC372Q, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.530573770 (1h29m37.31150715s) [noise=true , address=104.36.82.45:4934 , piece_id=ZZ6RPWOBUTIUH4HJO2F6BRSAIE4MX25MUAU33RBCGTXM2SWYJ56Q, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.530990045 (1h29m37.31192343s) [noise=true , address=79.127.219.38:4010 , piece_id=QM6SRZEJRYHYU27UHRSZXL3FQNEJTTAJHCEKZ6GAL7YWUFKQCAXA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.531330885 (1h29m37.312264267s) [noise=true , address=77.90.31.11:4579 , piece_id=DNJPW4C4I4NQAOZCMH7G7AU3R2T7D57EUGR6MU3K5UIEHSYVONYA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.531673876 (1h29m37.312607274s) [noise=true , address=104.36.82.49:4995 , piece_id=ZDH47LZTN4KE7UK4IVVCUH4BIKDEP2H3RHOI3QA6PNAGL5LVDRNA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.532146070 (1h29m37.313079452s) [noise=true , address=104.36.82.56:4257 , piece_id=BOOLF7NE75XUAX54VAV6MKZQ3GAXI2VB7WVMKYWMAG3CZQL4VMUQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.532352009 (1h29m37.313285394s) [noise=true , address=45.91.116.12:4230 , piece_id=BHLEAWCYA7RXYPN7RPLB27FWXZUDMJU6UIYX4RKY42OGJ6XF5HOQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.532745552 (1h29m37.313678937s) [noise=true , address=79.127.205.234:4619 , piece_id=5AIP2HHFQK24C5DXMOWDFY6HOZLNHSZIACC4UQW3RIHAYTWLA6GQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.533229557 (1h29m37.314162941s) [noise=true , address=79.127.219.51:4212 , piece_id=UC4GIFF6NJHZXXGICMWBWSSI6X7H7OLPJYNGNNBPXK4VUSDZYNZQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.533822806 (1h29m37.314756189s) [noise=true , address=104.36.82.55:4194 , piece_id=RRVNRK3A7PRECUAEK3IJFENPL4PA5JI7PALTWKLW26A5BOOVHXXQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.534204447 (1h29m37.315137837s) [noise=true , address=104.36.82.47:4548 , piece_id=75CSD3VG3BNCLBHLWDM3YCZLTMHOHNK6QNSJ3XUKVOF6NCVXSR2A, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.534696578 (1h29m37.315629944s) [noise=true , address=79.127.205.233:4640 , piece_id=XBZ5NQ3T6VLXZS6YVR5JVH5S576YWZ7KH7KVS4P3CVGIVA2SHKZA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.535168397 (1h29m37.31610178s) [noise=true , address=104.36.82.12:4181 , piece_id=5SXEVH354HFJ5Y7UKCRQLLWXX7YQL4KUKFLIF77ZS6HXORMCLXAA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.535841826 (1h29m37.316775212s) [noise=true , address=104.36.82.46:4513 , piece_id=NPWASIZDKGAPNK2N2WR4OMRLVPV64HGOWGITXVWZOBKVKELCJOOQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.615083883 (1h29m37.396017263s) [upload=1381]: Next returned nil segment Jul 26 23:21:21.615104244 (1h29m37.396037631s) [upload=1381]: Waiting for error group managing segments... Jul 26 23:21:21.656559331 (1h29m37.43749271s) [noise=true , address=77.90.31.11:4579 , piece_id=DNJPW4C4I4NQAOZCMH7G7AU3R2T7D57EUGR6MU3K5UIEHSYVONYA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.111769516 (1h29m37.892702894s) [noise=true , address=79.127.205.233:4640 , piece_id=XBZ5NQ3T6VLXZS6YVR5JVH5S576YWZ7KH7KVS4P3CVGIVA2SHKZA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.152805859 (1h29m37.933739226s) [noise=true , address=79.127.205.246:4285 , piece_id=Y65JV632XOGILLER2DN5YE3HFE2JGBGUO2D5OVOYIJAF3UBWMAUQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.153253815 (1h29m37.934187183s) [noise=true , address=79.127.205.234:4619 , piece_id=5AIP2HHFQK24C5DXMOWDFY6HOZLNHSZIACC4UQW3RIHAYTWLA6GQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.251001850 (1h29m38.031935228s) [noise=true , address=79.127.219.38:4010 , piece_id=QM6SRZEJRYHYU27UHRSZXL3FQNEJTTAJHCEKZ6GAL7YWUFKQCAXA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.259767917 (1h29m38.040701292s) [noise=true , address=45.91.116.12:4230 , piece_id=BHLEAWCYA7RXYPN7RPLB27FWXZUDMJU6UIYX4RKY42OGJ6XF5HOQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.504771339 (1h29m38.285704706s) [noise=true , address=156.146.43.225:4449 , piece_id=DJGEZPD5P64X2WIHWCRXALEHVEGR5NHPGRLMA5AJPDZMOYHSOC6A, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:23.678988341 (1h29m39.459921711s) [noise=true , address=79.127.219.51:4212 , piece_id=UC4GIFF6NJHZXXGICMWBWSSI6X7H7OLPJYNGNNBPXK4VUSDZYNZQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:22:21.359701382 (1h30m37.140634758s) [noise=true , address=104.36.82.46:4513 , piece_id=NPWASIZDKGAPNK2N2WR4OMRLVPV64HGOWGITXVWZOBKVKELCJOOQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.359772236 (1h30m37.140705611s) [noise=true , address=104.36.82.54:4183 , piece_id=SE4MN6GQKP2LIM3J4L7EBBC2MX5LOE2B5FZHQQ4L3Q7563M6Y5FQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.359828740 (1h30m37.140762115s) [noise=true , address=104.36.82.45:4934 , piece_id=ZZ6RPWOBUTIUH4HJO2F6BRSAIE4MX25MUAU33RBCGTXM2SWYJ56Q, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12TpurbQEmcBR8qoHKmAtxJAQinkzi8p5tX4Bn5i3UbDQ3kZHGT, address:104.36.82.45:4934): protocol: rpc: dial tcp 104.36.82.45:4934: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.359875499 (1h30m37.140808873s) [noise=true , address=77.90.31.10:4536 , piece_id=35F4NXUJ3NH7SU27OZAJPU2GWOACHL4AKAX7TNE2PCYZFLXVONHA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360136500 (1h30m37.141069875s) [noise=true , address=104.36.82.49:4995 , piece_id=ZDH47LZTN4KE7UK4IVVCUH4BIKDEP2H3RHOI3QA6PNAGL5LVDRNA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1F5oTP56RsDAA7Yde8hYvq7Lx5tiFDUHoBvPrYnhQiFy2Rko3N, address:104.36.82.49:4995): protocol: rpc: dial tcp 104.36.82.49:4995: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360236170 (1h30m37.141169546s) [noise=true , address=104.36.82.56:4257 , piece_id=BOOLF7NE75XUAX54VAV6MKZQ3GAXI2VB7WVMKYWMAG3CZQL4VMUQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:22:21.360310334 (1h30m37.14124371s) [noise=true , address=104.36.82.12:4181 , piece_id=5SXEVH354HFJ5Y7UKCRQLLWXX7YQL4KUKFLIF77ZS6HXORMCLXAA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: rpc: dial tcp 104.36.82.12:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360357676 (1h30m37.141291064s) [noise=true , address=104.36.82.55:4194 , piece_id=RRVNRK3A7PRECUAEK3IJFENPL4PA5JI7PALTWKLW26A5BOOVHXXQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: rpc: dial tcp 104.36.82.55:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360416360 (1h30m37.141349738s) [noise=true , address=104.36.82.47:4548 , piece_id=75CSD3VG3BNCLBHLWDM3YCZLTMHOHNK6QNSJ3XUKVOF6NCVXSR2A, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360432743 (1h30m37.141366119s) [noise=true , address=104.36.82.52:4120 , piece_id=MVPJLSLL65XNXKNEV5PPWGWZZFEX625WSK46RZPGBNX3D6UC372Q, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:42460->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360455657 (1h30m37.141389096s) [seg_pos={0 0}, upload=1381]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:22:21.394293706 (1h30m37.175227076s) [upload=1381]: Done uploading segments... Jul 26 23:22:21.394319252 (1h30m37.175252616s) [upload=1381]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.0365841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.043710063s FAIL (2025-07-26 23:21:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== (1h29m40.656875361s) [upload=1383]: Got next segment. Inline: false Jul 26 23:21:24.875986098 (1h29m40.656919484s) [upload=1383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:21:24.876111779 (1h29m40.657045154s) [upload=1383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:21:24.904134144 (1h29m40.68506753s) [seg_pos={0 0}, upload=1383]: Begin upload segment... Jul 26 23:21:24.904188340 (1h29m40.685121729s) [seg_pos={0 0}, upload=1383]: Done begin upload segment. Jul 26 23:21:24.904217544 (1h29m40.685151046s) [noise=true , address=104.36.82.7:4912 , piece_id=JNUNFLJZQ7WXSP5V7A3PJOEPSVKHBVWUZ7JRSMUGBENQT2Y5KL7Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.904475648 (1h29m40.685409029s) [noise=true , address=104.36.82.15:4258 , piece_id=B6RVNLJVLHBKIXPMV7CKCC6WTHFQOLFDGPM37R6IJ76EGFAAN37Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.904730497 (1h29m40.68566388s) [noise=true , address=45.91.116.7:4009 , piece_id=VXZG53SE2L3HCHTB3ONECGTCZHUH4I7PF2T7J5PLKU73GVENVRRA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905041032 (1h29m40.685974422s) [noise=true , address=104.36.82.13:4187 , piece_id=WZBPOFAYZ6TVOKQOUFYYKF3ZAXUGUG6YGQQE2FA4JF5Y5AUIRDHQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905200202 (1h29m40.686133583s) [noise=true , address=104.36.82.10:4077 , piece_id=LH7HSXRP5F3M27OOL22WRDJYQ4D6UU6QWGUC236CLQ52VHI6ACBQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905415045 (1h29m40.68634841s) [noise=true , address=104.36.82.16:4294 , piece_id=RQI6N65VRKGMQQX3PJD43M6CBFWEX2UDQBEO6FF6RVCRNADYGJEQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905661910 (1h29m40.686595293s) [noise=true , address=104.36.82.15:4259 , piece_id=ONBEFHTQVY5KNBDARLL7WNHG43J4FYT2EYHBWN6VQMO53F5SCZJA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905823721 (1h29m40.686757104s) [noise=true , address=45.91.116.8:4048 , piece_id=53LFZ3LUECBKFYH5T7DBSMQPJBQT2KFTQIYPZTDR2X66BIEMYTHA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906095107 (1h29m40.687028486s) [noise=true , address=45.91.116.4:4940 , piece_id=6UFARQPESYEZH4DT7A5OHKRXGPYBND7IOB4G2XXKM3YTZB6XHSIQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906224964 (1h29m40.687158342s) [noise=true , address=104.36.82.13:4186 , piece_id=RKOV6YNLKDATTPUOLI6TYIOKWDCLGVG5G63SPFOQCFZNIH5NB7GA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906521187 (1h29m40.687454569s) [noise=true , address=104.36.82.5:4840 , piece_id=MBPJZEIYDLTTDYRE4I2LAEGV6ITKQ73UWQFOLELVJQ3ZWFQO5S3Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906705673 (1h29m40.687639055s) [noise=true , address=104.36.82.16:4293 , piece_id=AGVINMCCQBK6UF6XVY5SBY7KNB3ISFMDEJL3KKQOCBO3RY6W36HA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906880616 (1h29m40.687813995s) [noise=true , address=104.36.82.6:4876 , piece_id=6K7WYV26EMCATYFGEMNXSFB6V65WLZXSDDDDLK743GJ4GCGMTYPQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.907361949 (1h29m40.688295333s) [noise=true , address=104.36.82.5:4837 , piece_id=W3XRFKCS7HBWFH4PQ4JAXLFBVKS6ADV73F5KEGZUFW3BHNN246IA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.907537001 (1h29m40.688470389s) [noise=true , address=45.91.116.10:4083 , piece_id=LH7SGQOJ2RPVDLDE7VWVSFGCLSA3LQS4VDFGA5X24JKHBOGM4R4Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.907952444 (1h29m40.688885824s) [noise=true , address=104.36.82.6:4874 , piece_id=UTFDFFBSZXICASY4WMKYS7VYDP6433XBHX2MYYZNVG5WH2VAS72A, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.987902788 (1h29m40.768836164s) [upload=1383]: Next returned nil segment Jul 26 23:21:24.987921368 (1h29m40.768854751s) [upload=1383]: Waiting for error group managing segments... Jul 26 23:21:24.992228344 (1h29m40.773161716s) [noise=true , address=45.91.116.4:4940 , piece_id=6UFARQPESYEZH4DT7A5OHKRXGPYBND7IOB4G2XXKM3YTZB6XHSIQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:24.992520053 (1h29m40.77345342s) [noise=true , address=45.91.116.8:4048 , piece_id=53LFZ3LUECBKFYH5T7DBSMQPJBQT2KFTQIYPZTDR2X66BIEMYTHA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:24.992813246 (1h29m40.773746623s) [noise=true , address=45.91.116.10:4083 , piece_id=LH7SGQOJ2RPVDLDE7VWVSFGCLSA3LQS4VDFGA5X24JKHBOGM4R4Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:24.993388771 (1h29m40.774322141s) [noise=true , address=45.91.116.7:4009 , piece_id=VXZG53SE2L3HCHTB3ONECGTCZHUH4I7PF2T7J5PLKU73GVENVRRA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:25.312949486 (1h29m41.093882869s) [noise=true , address=104.36.82.5:4840 , piece_id=MBPJZEIYDLTTDYRE4I2LAEGV6ITKQ73UWQFOLELVJQ3ZWFQO5S3Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:25.313201008 (1h29m41.094134376s) [noise=true , address=104.36.82.5:4837 , piece_id=W3XRFKCS7HBWFH4PQ4JAXLFBVKS6ADV73F5KEGZUFW3BHNN246IA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:54.845476978 (1h30m10.626410353s) [noise=true , address=104.36.82.13:4186 , piece_id=RKOV6YNLKDATTPUOLI6TYIOKWDCLGVG5G63SPFOQCFZNIH5NB7GA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845538046 (1h30m10.626471422s) [noise=true , address=104.36.82.16:4293 , piece_id=AGVINMCCQBK6UF6XVY5SBY7KNB3ISFMDEJL3KKQOCBO3RY6W36HA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845588026 (1h30m10.626521402s) [noise=true , address=104.36.82.16:4294 , piece_id=RQI6N65VRKGMQQX3PJD43M6CBFWEX2UDQBEO6FF6RVCRNADYGJEQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845632454 (1h30m10.626565829s) [noise=true , address=104.36.82.15:4259 , piece_id=ONBEFHTQVY5KNBDARLL7WNHG43J4FYT2EYHBWN6VQMO53F5SCZJA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845686829 (1h30m10.626620203s) [noise=true , address=104.36.82.6:4874 , piece_id=UTFDFFBSZXICASY4WMKYS7VYDP6433XBHX2MYYZNVG5WH2VAS72A, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:54.845765850 (1h30m10.626699226s) [noise=true , address=104.36.82.10:4077 , piece_id=LH7HSXRP5F3M27OOL22WRDJYQ4D6UU6QWGUC236CLQ52VHI6ACBQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:35210->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845814715 (1h30m10.62674809s) [noise=true , address=104.36.82.7:4912 , piece_id=JNUNFLJZQ7WXSP5V7A3PJOEPSVKHBVWUZ7JRSMUGBENQT2Y5KL7Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845849726 (1h30m10.626783104s) [noise=true , address=104.36.82.15:4258 , piece_id=B6RVNLJVLHBKIXPMV7CKCC6WTHFQOLFDGPM37R6IJ76EGFAAN37Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845866706 (1h30m10.626800082s) [noise=true , address=104.36.82.13:4187 , piece_id=WZBPOFAYZ6TVOKQOUFYYKF3ZAXUGUG6YGQQE2FA4JF5Y5AUIRDHQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845934911 (1h30m10.626868276s) [noise=true , address=104.36.82.6:4876 , piece_id=6K7WYV26EMCATYFGEMNXSFB6V65WLZXSDDDDLK743GJ4GCGMTYPQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845962141 (1h30m10.626895526s) [seg_pos={0 0}, upload=1383]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:21:54.887623879 (1h30m10.668557263s) [upload=1383]: Done uploading segments... Jul 26 23:21:54.887654984 (1h30m10.66858837s) [upload=1383]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.043728768s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........................................XXX.X....XX.X.X.X..] duration: 2.601413061s FAIL (2025-07-26 23:21:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.601439352s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........X........................X......XXXXXXXXXXXXXXXXXXX] duration: 30.037271059s FAIL (2025-07-26 23:21:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.037298598s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........................................XXX.X....XX.X.X.X.] duration: 18.537861762s FAIL (2025-07-26 23:21:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.537903398s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.034629034s FAIL (2025-07-26 23:21:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:20:19.696282789 (1h28m35.47721618s) [upload=1359]: Done uploading segments... Jul 26 23:20:19.696316054 (1h28m35.477249438s) [upload=1359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:20:54.780605082 (1h29m10.56153845s) [upload=1375]: Starting upload Jul 26 23:20:54.780685097 (1h29m10.561618473s) [upload=1375]: Done starting upload Jul 26 23:20:54.780873246 (1h29m10.561806619s) [upload=1375]: Uploading segments... Jul 26 23:20:54.780969346 (1h29m10.561902725s) [upload=1375]: Got next segment. Inline: false Jul 26 23:20:54.781023786 (1h29m10.561957151s) [upload=1375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:20:54.781030824 (1h29m10.561964191s) [upload=1375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:20:54.821098068 (1h29m10.60203146s) [seg_pos={0 0}, upload=1375]: Begin upload segment... Jul 26 23:20:54.821186209 (1h29m10.602119594s) [seg_pos={0 0}, upload=1375]: Done begin upload segment. Jul 26 23:20:54.821237776 (1h29m10.602171163s) [noise=true , address=45.91.116.14:4191 , piece_id=RHVNRXOHIFGUHA2XURBFDQESX4RN3LBQPH6Q4JS6VCKWFBWB2P5A, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.821251276 (1h29m10.602184665s) [noise=true , address=45.91.116.4:4939 , piece_id=5UM7Z3J4ZOGSX6F54HMY2F6QIMSDV5LZI5UG5KQEBXNXSN4TH2KA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.821836688 (1h29m10.602770068s) [noise=true , address=104.36.82.13:4186 , piece_id=LNN4L3K6DZT7PCAARPOBNLYUQCP4BDXPNLTPXQURDHEMS5EXC6YQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.821854541 (1h29m10.602787937s) [noise=true , address=45.91.116.12:4225 , piece_id=4HNJNZTM4CLGNVZDHUVAQQLLRPSLYU3JUPWZPWIC7UO766MANRCQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822245568 (1h29m10.603178956s) [noise=true , address=104.36.82.15:4258 , piece_id=UGSEOGCFCUPBF7UTFMOYDPSVHMYTCOU3OYEHQKYGENTYQEMOZWGA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822370180 (1h29m10.603303567s) [noise=true , address=104.36.82.9:4007 , piece_id=4DGXNTMQFYGGECEJ7MLPT5SQ3PLN6MMA2CIX4OZNVN5AV5DIGEAQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822642326 (1h29m10.603575712s) [noise=true , address=104.36.82.4:4803 , piece_id=BHWC2RH7U4BNVKWUZBXMKEXKQ456PP7PMQFOA72VWOIITW2ERMNA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822815078 (1h29m10.603748458s) [noise=true , address=104.36.82.5:4839 , piece_id=R42PIHSH7AU63AOEOPEKNECCO7JV7DIRZWAHXNCGGIALX3TOIWNA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823048692 (1h29m10.603982116s) [noise=true , address=45.91.116.8:4048 , piece_id=UXGDDX5SWKIK337JSNLNX74YQBYMFFDTMQDUDAXK63HSKHKPEQKA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823202570 (1h29m10.604135953s) [noise=true , address=45.91.116.8:4045 , piece_id=VFUJUZAY2N6K6PT4VDRUPXH62IWCIY6ND2UCI2M3OZ2DAV6FDZVQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823448779 (1h29m10.604382207s) [noise=true , address=104.36.82.4:4802 , piece_id=SPU4LB2HZI4HVO6CPICSLKZCSC2MVJWE5NZEAXTSTUNZ2QVIGSLQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823643543 (1h29m10.604576921s) [noise=true , address=45.91.116.6:4975 , piece_id=KLBSONQWZ3BP255JFMCJ2H3G7ZXWABOYPLX6WMRIGRXZK7YKX6DQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.824369551 (1h29m10.605302936s) [noise=true , address=104.36.82.7:4910 , piece_id=SXGCG2TV3ZB2T67GQVBR3M5WMGHPRAETDHYKDDEONEST35C26ERQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.825017700 (1h29m10.605951088s) [noise=true , address=104.36.82.2:4732 , piece_id=5AMJWB6CQ3HDRWXWGRDKSJCURN7VWYASAZU2U6HJVEFRZGQF3VGQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.825476169 (1h29m10.60640955s) [noise=true , address=77.90.31.11:4555 , piece_id=GLYBS57U43IH5R5A4UAFHPF7L5NLHNXJB7MDZMOKOYPXOHDKSS3A, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.825965990 (1h29m10.606899379s) [noise=true , address=45.91.116.7:4011 , piece_id=FTF4HD4SNSGMR5KFO2MKW2ABAXTZ4KBA6AQZTT4ZPTAWZPZ23U7Q, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.910664577 (1h29m10.691597945s) [upload=1375]: Next returned nil segment Jul 26 23:20:54.910695762 (1h29m10.691629141s) [upload=1375]: Waiting for error group managing segments... Jul 26 23:20:54.926409737 (1h29m10.707343107s) [noise=true , address=45.91.116.8:4048 , piece_id=UXGDDX5SWKIK337JSNLNX74YQBYMFFDTMQDUDAXK63HSKHKPEQKA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.926903994 (1h29m10.707837372s) [noise=true , address=45.91.116.6:4975 , piece_id=KLBSONQWZ3BP255JFMCJ2H3G7ZXWABOYPLX6WMRIGRXZK7YKX6DQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.927469880 (1h29m10.70840326s) [noise=true , address=45.91.116.8:4045 , piece_id=VFUJUZAY2N6K6PT4VDRUPXH62IWCIY6ND2UCI2M3OZ2DAV6FDZVQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.928199677 (1h29m10.709133051s) [noise=true , address=45.91.116.4:4939 , piece_id=5UM7Z3J4ZOGSX6F54HMY2F6QIMSDV5LZI5UG5KQEBXNXSN4TH2KA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.928674369 (1h29m10.709607743s) [noise=true , address=45.91.116.12:4225 , piece_id=4HNJNZTM4CLGNVZDHUVAQQLLRPSLYU3JUPWZPWIC7UO766MANRCQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.929086418 (1h29m10.710019799s) [noise=true , address=45.91.116.7:4011 , piece_id=FTF4HD4SNSGMR5KFO2MKW2ABAXTZ4KBA6AQZTT4ZPTAWZPZ23U7Q, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.963009092 (1h29m10.743942468s) [noise=true , address=77.90.31.11:4555 , piece_id=GLYBS57U43IH5R5A4UAFHPF7L5NLHNXJB7MDZMOKOYPXOHDKSS3A, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:55.197203553 (1h29m10.978136921s) [noise=true , address=45.91.116.14:4191 , piece_id=RHVNRXOHIFGUHA2XURBFDQESX4RN3LBQPH6Q4JS6VCKWFBWB2P5A, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:21:24.745983768 (1h29m40.526917133s) [noise=true , address=104.36.82.5:4839 , piece_id=R42PIHSH7AU63AOEOPEKNECCO7JV7DIRZWAHXNCGGIALX3TOIWNA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746050223 (1h29m40.526983588s) [noise=true , address=104.36.82.15:4258 , piece_id=UGSEOGCFCUPBF7UTFMOYDPSVHMYTCOU3OYEHQKYGENTYQEMOZWGA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746102931 (1h29m40.527036296s) [noise=true , address=104.36.82.4:4803 , piece_id=BHWC2RH7U4BNVKWUZBXMKEXKQ456PP7PMQFOA72VWOIITW2ERMNA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746153911 (1h29m40.527087274s) [noise=true , address=104.36.82.4:4802 , piece_id=SPU4LB2HZI4HVO6CPICSLKZCSC2MVJWE5NZEAXTSTUNZ2QVIGSLQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746211128 (1h29m40.527144503s) [noise=true , address=104.36.82.13:4186 , piece_id=LNN4L3K6DZT7PCAARPOBNLYUQCP4BDXPNLTPXQURDHEMS5EXC6YQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746265059 (1h29m40.527198421s) [noise=true , address=104.36.82.9:4007 , piece_id=4DGXNTMQFYGGECEJ7MLPT5SQ3PLN6MMA2CIX4OZNVN5AV5DIGEAQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746383434 (1h29m40.527316811s) [noise=true , address=104.36.82.7:4910 , piece_id=SXGCG2TV3ZB2T67GQVBR3M5WMGHPRAETDHYKDDEONEST35C26ERQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:24.746555461 (1h29m40.527488837s) [noise=true , address=104.36.82.2:4732 , piece_id=5AMJWB6CQ3HDRWXWGRDKSJCURN7VWYASAZU2U6HJVEFRZGQF3VGQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:24.746586951 (1h29m40.527520346s) [seg_pos={0 0}, upload=1375]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:21:24.778006617 (1h29m40.558939985s) [upload=1375]: Done uploading segments... Jul 26 23:21:24.778075045 (1h29m40.55900841s) [upload=1375]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.034733118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXX] duration: 1m0.032975824s FAIL (2025-07-26 23:21:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m37.06439314s) [upload=1366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:20:21.322547368 (1h28m37.103480733s) [seg_pos={0 0}, upload=1366]: Begin upload segment... Jul 26 23:20:21.322592790 (1h28m37.103526157s) [seg_pos={0 0}, upload=1366]: Done begin upload segment. Jul 26 23:20:21.322632798 (1h28m37.103566179s) [noise=true , address=104.36.82.54:4164 , piece_id=CMR5GRD5NZNE4TUABNZIMRD6WDH4WCWHYKZXRRQLMNKRHFAIEGSA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323122749 (1h28m37.10405613s) [noise=true , address=79.127.219.47:4061 , piece_id=ZYPHOABIF26IHVTA3XFY43AYX2JNRW4DGHFCKT27J42ZBG7ZBMUA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323260737 (1h28m37.104194124s) [noise=true , address=104.36.82.55:4201 , piece_id=QOU753O2MECZ5BRGQURSR56CKVTBWCUIDNGWBWL4NQALAGYSJK6Q, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323630846 (1h28m37.104564231s) [noise=true , address=77.90.31.10:4533 , piece_id=EOOAUUFLAU73UDM2DB54APPWPEMQZ2D3SHMOEY4TMSAAZBKONJ2Q, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323740940 (1h28m37.104674319s) [noise=true , address=104.36.82.41:4767 , piece_id=4CH2NP626LIIQK2LUBB67EYXO4WQ6LT2Z2IMQQILS5DTDLXPVIMQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324078417 (1h28m37.105011802s) [noise=true , address=104.36.82.47:4521 , piece_id=M46ALQQ2B3F6DMVURKWECLRTQXF4JO2QFDYI5H6B5ODYUDW5LVCA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324223156 (1h28m37.105156538s) [noise=true , address=79.127.201.215:4355 , piece_id=SIXB7SETQSE6D4A23P4IL27XP2OJ65YITSJODRQHF475YKYAL3PQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324458344 (1h28m37.105391729s) [noise=true , address=104.36.82.52:4096 , piece_id=6VR3FUBXLPSTRTJRGV2JOHD5O5JK76OOAYBOUF3ZNLPJGLIC2FHA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324654591 (1h28m37.105587983s) [noise=true , address=104.36.82.46:4490 , piece_id=FM7B3RPVGJAFSBPQBLYSGWZPAU4NOI6LAFYG5X2NVKT2ZIDP37ZQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324839447 (1h28m37.105772833s) [noise=true , address=64.49.13.170:4721 , piece_id=2UW3CM5JHWK6NQU7KBWVBXYGVF5RWKPH3WH3H7MXNW3OPRJLRMMQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325110176 (1h28m37.10604355s) [noise=true , address=77.90.31.15:4712 , piece_id=CS2J544LZFHBITY7YOH5FG6LWWZFHS6U4GULPUTIGIRI5UNGUYIA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325202354 (1h28m37.106135946s) [noise=true , address=77.90.31.11:4585 , piece_id=CDUWDVDXRQVCB5QR2UW5V2IZ6NSNELF5RPIE4WDWUCEE4PP6BZDA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325510983 (1h28m37.10644436s) [noise=true , address=64.49.13.140:4514 , piece_id=MUHEAX5ICBXCPGTYWPY634ZQZPSDGMLPN6URBGDD6YNRFKJG7ZHQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325584342 (1h28m37.106517725s) [noise=true , address=79.127.219.51:4210 , piece_id=3UAXVVDRK33TPOIWHNUUN7GIVS7XTY4BFMJ5KLENJBPI6QLSJ25A, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325847398 (1h28m37.106780775s) [noise=true , address=79.127.205.234:4603 , piece_id=VJAZYCAMJVZBUIMQWXICWUDLNAI3OS7MGXPAEOHDRCA763APB5CQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325947492 (1h28m37.106880876s) [noise=true , address=77.90.31.18:4809 , piece_id=YVEJ6DSHA7ARPP4MCANZHPSJLDZGZ3KXX3YXRBGEMURALS23KSAA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.326263032 (1h28m37.107196417s) [noise=true , address=79.127.205.245:4232 , piece_id=WS5DHVPBRDHOXCD2BAGQKCMJ6ID2QGBFQK6I2YDCLFWCQ6E5NGVA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.326332449 (1h28m37.107265831s) [noise=true , address=79.127.220.97:4361 , piece_id=WRNCBFABFZH36U4X6RK7RU2E37GFIER2SE3SKS347CLKI52AH3ZA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.405775038 (1h28m37.186708419s) [upload=1366]: Next returned nil segment Jul 26 23:20:21.405806965 (1h28m37.18674035s) [upload=1366]: Waiting for error group managing segments... Jul 26 23:20:21.464127395 (1h28m37.245060761s) [noise=true , address=77.90.31.18:4809 , piece_id=YVEJ6DSHA7ARPP4MCANZHPSJLDZGZ3KXX3YXRBGEMURALS23KSAA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:21.477125315 (1h28m37.258058682s) [noise=true , address=77.90.31.11:4585 , piece_id=CDUWDVDXRQVCB5QR2UW5V2IZ6NSNELF5RPIE4WDWUCEE4PP6BZDA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:21.524459841 (1h28m37.305393211s) [noise=true , address=79.127.220.97:4361 , piece_id=WRNCBFABFZH36U4X6RK7RU2E37GFIER2SE3SKS347CLKI52AH3ZA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:21.931563104 (1h28m37.712496471s) [noise=true , address=79.127.205.234:4603 , piece_id=VJAZYCAMJVZBUIMQWXICWUDLNAI3OS7MGXPAEOHDRCA763APB5CQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:22.030922152 (1h28m37.811855515s) [noise=true , address=64.49.13.170:4721 , piece_id=2UW3CM5JHWK6NQU7KBWVBXYGVF5RWKPH3WH3H7MXNW3OPRJLRMMQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:22.188677637 (1h28m37.969611007s) [noise=true , address=79.127.205.245:4232 , piece_id=WS5DHVPBRDHOXCD2BAGQKCMJ6ID2QGBFQK6I2YDCLFWCQ6E5NGVA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:23.911736832 (1h28m39.692670212s) [noise=true , address=79.127.219.51:4210 , piece_id=3UAXVVDRK33TPOIWHNUUN7GIVS7XTY4BFMJ5KLENJBPI6QLSJ25A, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:25.147147055 (1h28m40.928080421s) [noise=true , address=79.127.219.47:4061 , piece_id=ZYPHOABIF26IHVTA3XFY43AYX2JNRW4DGHFCKT27J42ZBG7ZBMUA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:25.624318759 (1h28m41.405252121s) [noise=true , address=77.90.31.15:4712 , piece_id=CS2J544LZFHBITY7YOH5FG6LWWZFHS6U4GULPUTIGIRI5UNGUYIA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:31.109755674 (1h28m46.890689043s) [noise=true , address=64.49.13.140:4514 , piece_id=MUHEAX5ICBXCPGTYWPY634ZQZPSDGMLPN6URBGDD6YNRFKJG7ZHQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:51.564486401 (1h29m7.345419769s) [noise=true , address=79.127.201.215:4355 , piece_id=SIXB7SETQSE6D4A23P4IL27XP2OJ65YITSJODRQHF475YKYAL3PQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:21:21.241425049 (1h29m37.022358425s) [noise=true , address=104.36.82.47:4521 , piece_id=M46ALQQ2B3F6DMVURKWECLRTQXF4JO2QFDYI5H6B5ODYUDW5LVCA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: write tcp 159.223.143.39:46802->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241508504 (1h29m37.022441881s) [noise=true , address=104.36.82.54:4164 , piece_id=CMR5GRD5NZNE4TUABNZIMRD6WDH4WCWHYKZXRRQLMNKRHFAIEGSA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:1gUXWffNccBoXWftiMWHHSTeEpA5zrd2cHz4uXX5j75GeEK9s2, address:104.36.82.54:4164): protocol: rpc: dial tcp 104.36.82.54:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241583453 (1h29m37.022516829s) [noise=true , address=104.36.82.55:4201 , piece_id=QOU753O2MECZ5BRGQURSR56CKVTBWCUIDNGWBWL4NQALAGYSJK6Q, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:21.241657158 (1h29m37.022590535s) [noise=true , address=77.90.31.10:4533 , piece_id=EOOAUUFLAU73UDM2DB54APPWPEMQZ2D3SHMOEY4TMSAAZBKONJ2Q, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: rpc: dial tcp 77.90.31.10:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241737771 (1h29m37.022671147s) [noise=true , address=104.36.82.41:4767 , piece_id=4CH2NP626LIIQK2LUBB67EYXO4WQ6LT2Z2IMQQILS5DTDLXPVIMQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug, address:104.36.82.41:4767): protocol: rpc: dial tcp 104.36.82.41:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241757118 (1h29m37.022690487s) [noise=true , address=104.36.82.52:4096 , piece_id=6VR3FUBXLPSTRTJRGV2JOHD5O5JK76OOAYBOUF3ZNLPJGLIC2FHA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241808976 (1h29m37.022742344s) [noise=true , address=104.36.82.46:4490 , piece_id=FM7B3RPVGJAFSBPQBLYSGWZPAU4NOI6LAFYG5X2NVKT2ZIDP37ZQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: write tcp 159.223.143.39:52118->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241852902 (1h29m37.022786271s) [seg_pos={0 0}, upload=1366]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:21:21.271746563 (1h29m37.052679932s) [upload=1366]: Done uploading segments... Jul 26 23:21:21.271775390 (1h29m37.052708754s) [upload=1366]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.03300394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [..........X........................X......XXXXXXXXXXXXXXXXXX] duration: 30.03165876s FAIL (2025-07-26 23:21:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.031683291s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................................XXX.X....XX.X.X.X] duration: 30.001109198s FAIL (2025-07-26 23:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001135816s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.................X.X..........XX...X...X..XX.X...X..X.......] duration: 16.616612562s FAIL (2025-07-26 23:20:44 UTC) context deadline exceeded Duration: 16.616661453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X.................X.X..........XX...X...X..XX.X...X..X......] duration: 44.472076741s FAIL (2025-07-26 23:20:27 UTC) context deadline exceeded Duration: 44.47209839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.03068359s FAIL (2025-07-26 23:20:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [upload=1359]: Got next segment. Inline: false Jul 26 23:19:49.705179586 (1h28m5.486112962s) [upload=1359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:19:49.705186345 (1h28m5.486119722s) [upload=1359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:19:49.737216056 (1h28m5.518149436s) [seg_pos={0 0}, upload=1359]: Begin upload segment... Jul 26 23:19:49.737324400 (1h28m5.518257789s) [noise=true , address=45.91.116.4:4937 , piece_id=32GPWGCNLK53GJJEGCHD4Y7IHBLCU5HEJWTUHGZFWQM5R5I4HTXA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.739175569 (1h28m5.520108958s) [seg_pos={0 0}, upload=1359]: Done begin upload segment. Jul 26 23:19:49.739262330 (1h28m5.520195711s) [noise=true , address=104.36.82.3:4765 , piece_id=AEWI22F3IXJV5XATZK4EZ5P3WSI637TQMOTKXPNULX3DMJOYJUYQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.740128738 (1h28m5.521062126s) [noise=true , address=104.36.82.14:4223 , piece_id=ZV5AKCPNALPANDEG4BZQH5QKGZ72JSPA4ZGTE4SRLWF55IY6SQ6Q, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.740738555 (1h28m5.521671936s) [noise=true , address=45.91.116.10:4081 , piece_id=OGOHUX5VKB27Y2MSGEMQX56SBV5FRG2MFRJ6D3ELOLYIWJ4TKVUA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.741476198 (1h28m5.522409581s) [noise=true , address=77.90.31.14:4661 , piece_id=Z7NP3MXFCD6BG3WDYLH5QAHZUDBXO64VGI3ZJK4L63DYNR4EHWNA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.742098185 (1h28m5.523031567s) [noise=true , address=104.36.82.8:4971 , piece_id=USF5JOGZMJ3SM5KCPYL3MLBRYGQIHHR3UPCHOVUHYRQPY7J7T32A, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.742604736 (1h28m5.523538123s) [noise=true , address=104.36.82.13:4185 , piece_id=YKJBH6MN36IFR4VUQP2O3GQLFUQAY6BRAC6JBSAEYU4RQNQ7D7JQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.743544502 (1h28m5.524477899s) [noise=true , address=45.91.116.13:4155 , piece_id=XBX54JV7XYGKQNKYSC3XD4M7OTI62WORHGJPEQZCR5J4TLHEPXPQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.744422133 (1h28m5.525355532s) [noise=true , address=104.36.82.9:4005 , piece_id=NF46AUDIQPJNS24YMGSQGZR4WOVZBJG3UAO24M4TZA3IX52BXLVA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.744992808 (1h28m5.525926196s) [noise=true , address=104.36.82.6:4875 , piece_id=P4FCC2AHDJKUCVJFATVZMA4RVUNKULSITDTTQN3VOIDUH4V4R7NA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.746722662 (1h28m5.527656052s) [noise=true , address=104.36.82.11:4114 , piece_id=TSGETJOBEYYHHPMZIPMVXGFOZ2XZXWIKF3YSKAQA3MJPN6PYH6YQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.747199309 (1h28m5.528132696s) [noise=true , address=104.36.82.5:4837 , piece_id=RARFXIZKLTG6FOFWAFYESPJA5BOQEVHY5YM4G54YFUXF67Z6FNBA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.747547213 (1h28m5.528480592s) [noise=true , address=45.91.116.13:4154 , piece_id=X4ZALKZ2DYEJ4NQCPSC7KUV7WIDESFURJHZ5WDELM5NGYWGOYWRQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.747926081 (1h28m5.528859461s) [noise=true , address=45.91.116.5:4299 , piece_id=B67XJC3Y74ZVW6G344CELCUFGB72E64H6GI5HQ2UALWSJ7AACV4A, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.748331977 (1h28m5.52926537s) [noise=true , address=104.36.82.10:4079 , piece_id=EUQTUOWX66CFSX2ZGOBAEPYDJ7I3IRJXMA4EUX2EMEFUX6LIUDJQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.748745773 (1h28m5.52967916s) [noise=true , address=104.36.82.4:4803 , piece_id=DXNYZOV6KI7BDE34WGVDGFHWJMT3YZG3AXLNPH342VW6W6FAP76A, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.838319936 (1h28m5.619253314s) [upload=1359]: Next returned nil segment Jul 26 23:19:49.838329763 (1h28m5.619263128s) [upload=1359]: Waiting for error group managing segments... Jul 26 23:19:49.842535778 (1h28m5.623469145s) [noise=true , address=45.91.116.13:4155 , piece_id=XBX54JV7XYGKQNKYSC3XD4M7OTI62WORHGJPEQZCR5J4TLHEPXPQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.842933898 (1h28m5.623867261s) [noise=true , address=45.91.116.10:4081 , piece_id=OGOHUX5VKB27Y2MSGEMQX56SBV5FRG2MFRJ6D3ELOLYIWJ4TKVUA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.843423189 (1h28m5.624356566s) [noise=true , address=45.91.116.5:4299 , piece_id=B67XJC3Y74ZVW6G344CELCUFGB72E64H6GI5HQ2UALWSJ7AACV4A, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.843876736 (1h28m5.624810098s) [noise=true , address=45.91.116.4:4937 , piece_id=32GPWGCNLK53GJJEGCHD4Y7IHBLCU5HEJWTUHGZFWQM5R5I4HTXA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.844397457 (1h28m5.625330839s) [noise=true , address=77.90.31.14:4661 , piece_id=Z7NP3MXFCD6BG3WDYLH5QAHZUDBXO64VGI3ZJK4L63DYNR4EHWNA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.845034068 (1h28m5.625967429s) [noise=true , address=45.91.116.13:4154 , piece_id=X4ZALKZ2DYEJ4NQCPSC7KUV7WIDESFURJHZ5WDELM5NGYWGOYWRQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:20:19.668998314 (1h28m35.449931693s) [noise=true , address=104.36.82.13:4185 , piece_id=YKJBH6MN36IFR4VUQP2O3GQLFUQAY6BRAC6JBSAEYU4RQNQ7D7JQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669096474 (1h28m35.450029862s) [noise=true , address=104.36.82.9:4005 , piece_id=NF46AUDIQPJNS24YMGSQGZR4WOVZBJG3UAO24M4TZA3IX52BXLVA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669186252 (1h28m35.45011964s) [noise=true , address=104.36.82.10:4079 , piece_id=EUQTUOWX66CFSX2ZGOBAEPYDJ7I3IRJXMA4EUX2EMEFUX6LIUDJQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669266337 (1h28m35.4501997s) [noise=true , address=104.36.82.14:4223 , piece_id=ZV5AKCPNALPANDEG4BZQH5QKGZ72JSPA4ZGTE4SRLWF55IY6SQ6Q, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669356614 (1h28m35.450289993s) [noise=true , address=104.36.82.8:4971 , piece_id=USF5JOGZMJ3SM5KCPYL3MLBRYGQIHHR3UPCHOVUHYRQPY7J7T32A, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669427226 (1h28m35.450360606s) [noise=true , address=104.36.82.6:4875 , piece_id=P4FCC2AHDJKUCVJFATVZMA4RVUNKULSITDTTQN3VOIDUH4V4R7NA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669515013 (1h28m35.450448391s) [noise=true , address=104.36.82.11:4114 , piece_id=TSGETJOBEYYHHPMZIPMVXGFOZ2XZXWIKF3YSKAQA3MJPN6PYH6YQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669580172 (1h28m35.450513551s) [noise=true , address=104.36.82.4:4803 , piece_id=DXNYZOV6KI7BDE34WGVDGFHWJMT3YZG3AXLNPH342VW6W6FAP76A, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669642835 (1h28m35.450576218s) [noise=true , address=104.36.82.3:4765 , piece_id=AEWI22F3IXJV5XATZK4EZ5P3WSI637TQMOTKXPNULX3DMJOYJUYQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669710891 (1h28m35.450644274s) [noise=true , address=104.36.82.5:4837 , piece_id=RARFXIZKLTG6FOFWAFYESPJA5BOQEVHY5YM4G54YFUXF67Z6FNBA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:48088->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669734180 (1h28m35.450667559s) [seg_pos={0 0}, upload=1359]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:20:19.696282789 (1h28m35.47721618s) [upload=1359]: Done uploading segments... Jul 26 23:20:19.696316054 (1h28m35.477249438s) [upload=1359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.030709815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXX] duration: 1m0.027311101s FAIL (2025-07-26 23:20:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== ce... Jul 26 23:19:18.135975537 (1h27m33.916908922s) [noise=true , address=79.127.219.47:4048 , piece_id=P2K46E5XGOIYAFSHSRJF3JY6HSPKHED6J764BMLUM7WNWKBX7POA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.137137960 (1h27m33.918071354s) [noise=true , address=45.91.116.9:4267 , piece_id=LVGE5PKTAQXWUKRIEKPDT7YT4CNCQXPKAITMUR4CVDKR4CLLIOFQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.137513910 (1h27m33.918447295s) [noise=true , address=104.36.82.52:4103 , piece_id=NDEAO2NKTJSZVSSJPDYBX7DIAW5FXKNPW465M6P3FPK27ERWJVQA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.139192427 (1h27m33.920125811s) [noise=true , address=104.36.82.12:4159 , piece_id=X6SSZ222NHB5K4T45H3YLTAF3N35R6DZCWZCRNWHVF7QPIGYXWWQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.139623129 (1h27m33.920556507s) [noise=true , address=77.90.31.17:4791 , piece_id=DBZUJY3YKUU2PFQLSSHFKMY74Z6MTFBH7R6YGL4N7B5KXZA3WKSA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.139911553 (1h27m33.920844931s) [noise=true , address=104.36.82.56:4244 , piece_id=2VJ34FX62U5PKA7KQG7LJXEEOJSGWIGU77CYHS7GBHYTWA4Q6JEQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.140405210 (1h27m33.921338591s) [noise=true , address=104.36.82.46:4508 , piece_id=OXUNIGFQJUAVCWGU5QVYCDAJGVI5LAN22YIZU2VEX6Y737YJBMZQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.140790322 (1h27m33.921723814s) [noise=true , address=45.91.116.14:4206 , piece_id=ABMV2VQWYVR3FYW34RFGL4F42BLY7C2PXACAWYKNBNSTVRPSYUXQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.141277180 (1h27m33.922210566s) [noise=true , address=45.91.116.9:4288 , piece_id=XOL24QH2GGVV44AAX47WANUU3Z6NALHYWMROFEZEAGT2MI32ICLQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.141569335 (1h27m33.922502712s) [noise=true , address=64.49.13.138:4482 , piece_id=W7BRVGGVKCQZ46NBWOJ62MQ7ZVHVECKP2I5DPFUQOZ2QVCFBWPOQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.142943189 (1h27m33.923876568s) [noise=true , address=77.90.31.11:4564 , piece_id=HYWRCLNSHRNGAI5VQVZDHTOSTWYYU2ADQHC4LZWBNP2QZRT3QILA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.207820665 (1h27m33.988754048s) [upload=1351]: Next returned nil segment Jul 26 23:19:18.207842675 (1h27m33.988776061s) [upload=1351]: Waiting for error group managing segments... Jul 26 23:19:18.280278539 (1h27m34.061211907s) [noise=true , address=77.90.31.11:4564 , piece_id=HYWRCLNSHRNGAI5VQVZDHTOSTWYYU2ADQHC4LZWBNP2QZRT3QILA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:19:19.069249385 (1h27m34.850182755s) [noise=true , address=64.49.13.138:4482 , piece_id=W7BRVGGVKCQZ46NBWOJ62MQ7ZVHVECKP2I5DPFUQOZ2QVCFBWPOQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:19:20.402617511 (1h27m36.183550897s) [noise=true , address=45.91.116.14:4206 , piece_id=ABMV2VQWYVR3FYW34RFGL4F42BLY7C2PXACAWYKNBNSTVRPSYUXQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:19:21.183947592 (1h27m36.96488096s) [noise=true , address=79.127.219.47:4048 , piece_id=P2K46E5XGOIYAFSHSRJF3JY6HSPKHED6J764BMLUM7WNWKBX7POA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:19:23.305968756 (1h27m39.086902122s) [noise=true , address=45.91.116.9:4267 , piece_id=LVGE5PKTAQXWUKRIEKPDT7YT4CNCQXPKAITMUR4CVDKR4CLLIOFQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:19:24.350400852 (1h27m40.131334215s) [noise=true , address=45.91.116.9:4288 , piece_id=XOL24QH2GGVV44AAX47WANUU3Z6NALHYWMROFEZEAGT2MI32ICLQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:20:18.058658937 (1h28m33.839592318s) [noise=true , address=77.90.31.19:4844 , piece_id=Z5GK5IM35AKKL52JWQTT3GU5FR3L7LX4IKIBCPD3LRT3EMBVNBAQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:20:18.058923493 (1h28m33.839856879s) [noise=true , address=104.36.82.46:4508 , piece_id=OXUNIGFQJUAVCWGU5QVYCDAJGVI5LAN22YIZU2VEX6Y737YJBMZQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.058994842 (1h28m33.839928229s) [noise=true , address=104.36.82.47:4552 , piece_id=FULWZTWXU44ZZWTW76JDV25PGHTPJ7FA5NE4ZXY62OD6DLC77H4Q, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059093724 (1h28m33.840027091s) [noise=true , address=104.36.82.7:4922 , piece_id=ZP4XY4QZCOKISAC4ZU6GHM3PCJJBGPBVGO2EINHVQ4RSJPJCACMQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12KrMgT4Pwj9UFE88q6tt65D7o6q68NhsmEXmifNmvzJ6eEkm8R, address:104.36.82.7:4922): protocol: write tcp 159.223.143.39:51316->104.36.82.7:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059347392 (1h28m33.840280773s) [noise=true , address=104.36.82.54:4158 , piece_id=XGB5B7N5R52VNDERB5UAEI5MLQTY54ZESM337E2WHV6FCYCAKPQA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: write tcp 159.223.143.39:54914->104.36.82.54:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059429669 (1h28m33.840363038s) [noise=true , address=104.36.82.52:4103 , piece_id=NDEAO2NKTJSZVSSJPDYBX7DIAW5FXKNPW465M6P3FPK27ERWJVQA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 159.223.143.39:45784->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059509324 (1h28m33.840442701s) [noise=true , address=77.90.31.17:4791 , piece_id=DBZUJY3YKUU2PFQLSSHFKMY74Z6MTFBH7R6YGL4N7B5KXZA3WKSA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: write tcp 159.223.143.39:52948->77.90.31.17:4791: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059650251 (1h28m33.840583629s) [noise=true , address=104.36.82.55:4206 , piece_id=EPMP6WI5FKTXEOFMU4G2C5WOXOSDNU5QPOCVKZ7JQIWLRM5J7PIA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: write tcp 159.223.143.39:56182->104.36.82.55:4206: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059714220 (1h28m33.840647585s) [noise=true , address=104.36.82.53:4147 , piece_id=6OTNFOAJWQYOLRAXPUZEZUMT2H7IQ6EET54L5J3ZCRZUSHAMGJZA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 159.223.143.39:58138->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059776542 (1h28m33.840709921s) [noise=true , address=104.36.82.12:4159 , piece_id=X6SSZ222NHB5K4T45H3YLTAF3N35R6DZCWZCRNWHVF7QPIGYXWWQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: write tcp 159.223.143.39:59372->104.36.82.12:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059787757 (1h28m33.840721141s) [noise=true , address=104.36.82.56:4244 , piece_id=2VJ34FX62U5PKA7KQG7LJXEEOJSGWIGU77CYHS7GBHYTWA4Q6JEQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 159.223.143.39:46874->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059827230 (1h28m33.840760612s) [noise=true , address=77.90.31.16:4761 , piece_id=M7D5SOI7JKUKFFNCOMTKMZN2DCA6QSQAP6BPIUW3NTDLPNCEGS2Q, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:48900->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059860232 (1h28m33.84079362s) [seg_pos={0 0}, upload=1351]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 23:20:18.083219670 (1h28m33.86415304s) [upload=1351]: Done uploading segments... Jul 26 23:20:18.083280907 (1h28m33.864214287s) [upload=1351]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.027329753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 19 out of 60 is failed (threshold=3) [...........X........................X......XXXXXXXXXXXXXXXXX] duration: 30.031355956s FAIL (2025-07-26 23:20:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.031386749s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................XXX.X....XX.X.X.] duration: 23.880314367s FAIL (2025-07-26 23:19:50 UTC) orbiter warning: context deadline exceeded Duration: 23.880345791s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XX.................X.X..........XX...X...X..XX.X...X..X.....] duration: 55.627074274s FAIL (2025-07-26 23:19:43 UTC) context deadline exceeded Duration: 55.627110065s core uplink us-select-1 us1-placement-20

1
...
6763
6764
6765
6766
6767
...
7257