Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 1m0.027324858s FAIL (2025-07-26 22:23:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== (31m7.054824011s) [seg_pos={0 0}, upload=484]: Done begin upload segment. Jul 26 22:22:51.273919256 (31m7.054852636s) [noise=true , address=104.36.82.21:4501 , piece_id=6RAHEJCE4642E4AWNDZI26LE6TRXMD2OKLYYC5C5MM7HG5LA7V6A, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.274314475 (31m7.055247842s) [noise=true , address=104.36.82.40:4947 , piece_id=LDZOS3OEHIC7VO3SYKMCZI65UR2KRJO3JLAKE36Q4J6DIMP4TV7Q, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.274350819 (31m7.055284224s) [noise=true , address=79.127.205.245:4251 , piece_id=BYWGI4FDZGTVUSN7HAHIJID2Y4HQJHCV7ZGGIMV6VVTOAFBZZDCQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.274793936 (31m7.055727324s) [noise=true , address=104.36.82.35:4599 , piece_id=BAJFCY5TWH4PTLW36UMP4YHOBQONEBQZ7TSMCSHI5XUJOAMJHSHQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.274888331 (31m7.055821708s) [noise=true , address=79.127.205.235:4668 , piece_id=JFE7YOCBNBWB2FF3YKBAWRJDV67ZZHRUOEDQW27CVHFXK4NBWH7A, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.275269601 (31m7.056203008s) [noise=true , address=79.127.219.47:4061 , piece_id=Z6343665MIEPYIELYEMXVQSQ6T5VIXUTF5FLOFWRWYCGBBMY6KAA, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.275467883 (31m7.056401264s) [noise=true , address=104.36.82.4:4824 , piece_id=UMGTNL7PJWIC72AQLRYE4TKF6J32TULVF72JTSX43PSM7ZZ7TCXQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.275625386 (31m7.056558765s) [noise=true , address=79.127.219.38:4009 , piece_id=D6ILOBRCMDEMOV5M24CKLU5SM7MPWAJ47H6FYIWK3ZA7MWYAVKWQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.275948096 (31m7.056881473s) [noise=true , address=156.146.43.225:4448 , piece_id=DZYMPPJEUC7A3MTDZ52GZDAYRJ2SP6Z6CFFWWKWJ2NIN2SPVPDXA, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.275959672 (31m7.056893055s) [noise=true , address=104.36.82.56:4230 , piece_id=HUU6VPRPTBLK6OTOJGYM7H2ICMCE4D3VSFDTVF6V2VKRD2W7YKAQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.276422199 (31m7.057355581s) [noise=true , address=77.90.31.16:4737 , piece_id=SHN5KS7VJNUFCZF67SYAMXZAHK3N3U7YENRVSOSYWRYLJN3PCNGQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.276593792 (31m7.057527187s) [noise=true , address=79.127.205.234:4620 , piece_id=GQB3AYLCYQ327AEAMTZZBIVL3PJTCTA4LSBKSMDYKWDT6YMSUSJQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.277114519 (31m7.058047907s) [noise=true , address=104.36.82.42:4824 , piece_id=AJ7OFMYPEAELCURKWN5CSRJI6EXJNZMRUDKX2LITUPKZVGWQZGTA, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.278721869 (31m7.059655257s) [noise=true , address=79.127.201.215:4357 , piece_id=SX53P7GNLCX5PTT4GUP6OMFAUT5I66XSYKMAXJ4GISBBC6YAW4XA, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.279226235 (31m7.060159624s) [noise=true , address=79.127.205.246:4286 , piece_id=RBKFWDFABPRDDIS6UJOSE3WFQMA5BEEJEKKHIBFHUKHBJPN724VA, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.279669492 (31m7.06060288s) [noise=true , address=45.91.116.9:4269 , piece_id=XUL3BEIDSOD7UDJQKZRIC6D3GB56DLNEIVQ57VOKPZ4UJOAWB7DQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.280059004 (31m7.060992368s) [noise=true , address=104.36.82.7:4919 , piece_id=Y2T5GWGWIGTADUMKTTWIRYHP4HNO6DRRGYV7ANL7YFGNKYWJUBCQ, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.284430637 (31m7.065363999s) [noise=true , address=77.90.31.10:4540 , piece_id=RGXFU4ITYQ3AXOWW5RNGNHOF6EEH2U7FY3NCHJP3P73B5I7VNY7A, seg_pos={0 0}, upload=484]: Uploading piece... Jul 26 22:22:51.335427546 (31m7.116360939s) [noise=true , address=79.127.219.47:4061 , piece_id=Z6343665MIEPYIELYEMXVQSQ6T5VIXUTF5FLOFWRWYCGBBMY6KAA, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:22:51.356721776 (31m7.137655139s) [upload=484]: Next returned nil segment Jul 26 22:22:51.356742354 (31m7.137675743s) [upload=484]: Waiting for error group managing segments... Jul 26 22:22:51.382849606 (31m7.16378297s) [noise=true , address=79.127.219.38:4009 , piece_id=D6ILOBRCMDEMOV5M24CKLU5SM7MPWAJ47H6FYIWK3ZA7MWYAVKWQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:22:51.426164866 (31m7.207098233s) [noise=true , address=156.146.43.225:4448 , piece_id=DZYMPPJEUC7A3MTDZ52GZDAYRJ2SP6Z6CFFWWKWJ2NIN2SPVPDXA, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:22:51.462772123 (31m7.243705492s) [noise=true , address=77.90.31.10:4540 , piece_id=RGXFU4ITYQ3AXOWW5RNGNHOF6EEH2U7FY3NCHJP3P73B5I7VNY7A, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:22:51.481351007 (31m7.262284387s) [noise=true , address=77.90.31.16:4737 , piece_id=SHN5KS7VJNUFCZF67SYAMXZAHK3N3U7YENRVSOSYWRYLJN3PCNGQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:22:51.876750658 (31m7.657684021s) [noise=true , address=79.127.205.234:4620 , piece_id=GQB3AYLCYQ327AEAMTZZBIVL3PJTCTA4LSBKSMDYKWDT6YMSUSJQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:22:51.912522008 (31m7.693455371s) [noise=true , address=79.127.205.246:4286 , piece_id=RBKFWDFABPRDDIS6UJOSE3WFQMA5BEEJEKKHIBFHUKHBJPN724VA, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:22:51.923384348 (31m7.704317733s) [noise=true , address=79.127.205.235:4668 , piece_id=JFE7YOCBNBWB2FF3YKBAWRJDV67ZZHRUOEDQW27CVHFXK4NBWH7A, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:22:52.866217550 (31m8.64715093s) [noise=true , address=79.127.205.245:4251 , piece_id=BYWGI4FDZGTVUSN7HAHIJID2Y4HQJHCV7ZGGIMV6VVTOAFBZZDCQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:22:58.235583608 (31m14.016516972s) [noise=true , address=45.91.116.9:4269 , piece_id=XUL3BEIDSOD7UDJQKZRIC6D3GB56DLNEIVQ57VOKPZ4UJOAWB7DQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:23:02.592890219 (31m18.373823596s) [noise=true , address=79.127.201.215:4357 , piece_id=SX53P7GNLCX5PTT4GUP6OMFAUT5I66XSYKMAXJ4GISBBC6YAW4XA, seg_pos={0 0}, upload=484]: Done uploading piece. err: <nil> Jul 26 22:23:51.202745362 (32m6.983678745s) [noise=true , address=104.36.82.21:4501 , piece_id=6RAHEJCE4642E4AWNDZI26LE6TRXMD2OKLYYC5C5MM7HG5LA7V6A, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: rpc: dial tcp 104.36.82.21:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:51.202827401 (32m6.983760778s) [noise=true , address=104.36.82.35:4599 , piece_id=BAJFCY5TWH4PTLW36UMP4YHOBQONEBQZ7TSMCSHI5XUJOAMJHSHQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:1nxKQjpzyhZsRnkffX1fwCXcTYiwHPwuNg4HWtpdzRnKQCtp6s, address:104.36.82.35:4599): protocol: write tcp 159.223.143.39:46588->104.36.82.35:4599: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:51.202922028 (32m6.983855408s) [noise=true , address=104.36.82.42:4824 , piece_id=AJ7OFMYPEAELCURKWN5CSRJI6EXJNZMRUDKX2LITUPKZVGWQZGTA, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:12QTjnomQYFETHbaJBWxWEPo5yweArk68bKPeooZXpRZrpw1ACq, address:104.36.82.42:4824): protocol: rpc: dial tcp 104.36.82.42:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:51.202989727 (32m6.983923093s) [noise=true , address=104.36.82.4:4824 , piece_id=UMGTNL7PJWIC72AQLRYE4TKF6J32TULVF72JTSX43PSM7ZZ7TCXQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:51.203096949 (32m6.984030327s) [noise=true , address=104.36.82.40:4947 , piece_id=LDZOS3OEHIC7VO3SYKMCZI65UR2KRJO3JLAKE36Q4J6DIMP4TV7Q, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:104.36.82.40:4947): protocol: rpc: dial tcp 104.36.82.40:4947: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:51.203138512 (32m6.984071893s) [noise=true , address=104.36.82.56:4230 , piece_id=HUU6VPRPTBLK6OTOJGYM7H2ICMCE4D3VSFDTVF6V2VKRD2W7YKAQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:51.203200262 (32m6.98413363s) [noise=true , address=104.36.82.7:4919 , piece_id=Y2T5GWGWIGTADUMKTTWIRYHP4HNO6DRRGYV7ANL7YFGNKYWJUBCQ, seg_pos={0 0}, upload=484]: Done uploading piece. err: ecclient: upload failed (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w, address:104.36.82.7:4919): protocol: write tcp 159.223.143.39:55714->104.36.82.7:4919: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:51.203244514 (32m6.984177902s) [seg_pos={0 0}, upload=484]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:23:51.226193204 (32m7.007126572s) [upload=484]: Done uploading segments... Jul 26 22:23:51.226219135 (32m7.007152512s) [upload=484]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.027348739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 572.981316ms FAIL (2025-07-26 22:23:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 573.019345ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................XX..XXX..X..XX..X..XX.X....X....X.......] duration: 594.039221ms FAIL (2025-07-26 22:23:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 594.070538ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 6.074078872s FAIL (2025-07-26 22:23:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.074096462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 1m0.00109336s FAIL (2025-07-26 22:23:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001117281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [................X..X........XXX.XXX.X.XX..X..X......X.......] duration: 800.360561ms FAIL (2025-07-26 22:23:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 800.397991ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 504.794155ms FAIL (2025-07-26 22:22:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.818818ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.................X..X........XXX.XXX.X.XX..X..X......X......] duration: 650.514581ms FAIL (2025-07-26 22:22:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 650.547885ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 1m0.040540809s FAIL (2025-07-26 22:22:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 17441s) [noise=true , address=104.36.82.10:4090 , piece_id=26Y24VLIT4PILDM7PQQG5ZGG3T6O6KRKECW3W3FLQFYYID3RVBIA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.184699962 (30m6.965633351s) [noise=true , address=104.36.82.4:4823 , piece_id=TH253XT6ANY3TQHCZPYHXXRXNEYTB5GFWOYAQKUPSOZK4RIQVBDQ, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.185193148 (30m6.966126533s) [noise=true , address=79.127.205.233:4642 , piece_id=AMWVZE4HYV3MC7MW4WNYBFEINBNEI3YDNGZLKNWYZDBZAFEIBNAA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.185363737 (30m6.966297122s) [noise=true , address=45.91.116.10:4109 , piece_id=5LILZ4HXA34GMSMWUOA3ETQGUAMQXM5DEYBGKLFULI5QANHYETYA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.185593219 (30m6.966526596s) [noise=true , address=104.36.82.12:4164 , piece_id=SKNQD6JQQET7CTI6OU5FSA2WUECERJPMHKE34SHW5RPFANWML2PQ, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.185963556 (30m6.966896948s) [noise=true , address=104.36.82.2:4751 , piece_id=YKHFYDOUVJKIIX3FYWAS23AMQP7HDLB4HWNPB2KUHWOAYCZOYTWA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.186026930 (30m6.966960318s) [noise=true , address=79.127.205.234:4604 , piece_id=J7DHYPBF3254AY3FWXLEBFC2HBUDK65YAVTLAPVDRS3Y5UDXJQHQ, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.186467411 (30m6.967400805s) [noise=true , address=104.36.82.41:4795 , piece_id=FUOCDY3HJEZHJMDLFJCIF72XWWXCBHVS2AMWCD5L4FRBMJ5S4QCQ, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.186512495 (30m6.967445883s) [noise=true , address=45.91.116.5:4310 , piece_id=SLXTD6BCMIYTGCSGZ5VQMRBFZSX665ZRGRQDENEAWGZBXFGJ3MCA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.186556996 (30m6.967490379s) [noise=true , address=104.36.82.11:4147 , piece_id=SDXVXM5HBHOF4TD7FL5SXUSNMELHZX5KHPXKDXZOCX7QGJ4DMEEQ, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.186604778 (30m6.967538168s) [noise=true , address=79.127.205.235:4681 , piece_id=CF3FACMS2VIAVR55QVTAAAMROUX37DYCPTBPDJXULHOY2VCI4RAQ, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.187120598 (30m6.968053991s) [noise=true , address=104.36.82.13:4193 , piece_id=X4H7E5467UIBFY6YJUNHWCYJ6JT33OR6BPV3FHZTEHDJPUNST7OA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.187611542 (30m6.968544954s) [noise=true , address=104.36.82.34:4551 , piece_id=4IOL6UNVF3LW5G5JWLNJUEOVMMNKD4F7CBJJ5NXSD2BQYSXFPEJA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.188026260 (30m6.968959649s) [noise=true , address=79.127.219.38:4006 , piece_id=M3PCV7UVJNBFBU4WHP5OL5TKELC3CNI3HXI2T2ZSFXNGDP72QAHQ, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.188535095 (30m6.969468486s) [noise=true , address=79.127.219.47:4069 , piece_id=4PA74IN3DPGTYGFFILENZWCRXKOLTT2BZMSVPB4KEEY2GE2HZQOA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.189500593 (30m6.970433981s) [noise=true , address=104.36.82.19:4408 , piece_id=3G3ENNDUYJV2BPFQ7Q7P54LGRPZRWEY5SFIKNHKEE6XOXOJ4TBBA, seg_pos={0 0}, upload=471]: Uploading piece... Jul 26 22:21:51.291991627 (30m7.072925004s) [upload=471]: Next returned nil segment Jul 26 22:21:51.292009872 (30m7.072943235s) [upload=471]: Waiting for error group managing segments... Jul 26 22:21:51.365420357 (30m7.146353742s) [noise=true , address=156.146.43.225:4448 , piece_id=BHF3X54KACJBCW7T3KMGOAS4PMMB4RPHG6RY5PMKREZGIZWSLFYA, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:21:51.542892862 (30m7.323826241s) [noise=true , address=79.127.219.38:4006 , piece_id=M3PCV7UVJNBFBU4WHP5OL5TKELC3CNI3HXI2T2ZSFXNGDP72QAHQ, seg_pos={0 0}, upload=471]: Done uploading piece. err: <nil> Jul 26 22:21:51.775518940 (30m7.55645233s) [noise=true , address=79.127.205.234:4604 , piece_id=J7DHYPBF3254AY3FWXLEBFC2HBUDK65YAVTLAPVDRS3Y5UDXJQHQ, seg_pos={0 0}, upload=471]: Done uploading piece. err: <nil> Jul 26 22:21:51.800684718 (30m7.581618106s) [noise=true , address=79.127.205.233:4642 , piece_id=AMWVZE4HYV3MC7MW4WNYBFEINBNEI3YDNGZLKNWYZDBZAFEIBNAA, seg_pos={0 0}, upload=471]: Done uploading piece. err: <nil> Jul 26 22:21:51.828294998 (30m7.609228363s) [noise=true , address=79.127.205.235:4681 , piece_id=CF3FACMS2VIAVR55QVTAAAMROUX37DYCPTBPDJXULHOY2VCI4RAQ, seg_pos={0 0}, upload=471]: Done uploading piece. err: <nil> Jul 26 22:21:52.485816878 (30m8.266750241s) [noise=true , address=45.91.116.10:4109 , piece_id=5LILZ4HXA34GMSMWUOA3ETQGUAMQXM5DEYBGKLFULI5QANHYETYA, seg_pos={0 0}, upload=471]: Done uploading piece. err: <nil> Jul 26 22:21:52.844661810 (30m8.625595186s) [noise=true , address=79.127.219.47:4069 , piece_id=4PA74IN3DPGTYGFFILENZWCRXKOLTT2BZMSVPB4KEEY2GE2HZQOA, seg_pos={0 0}, upload=471]: Done uploading piece. err: <nil> Jul 26 22:21:53.477331018 (30m9.258264402s) [noise=true , address=45.91.116.5:4310 , piece_id=SLXTD6BCMIYTGCSGZ5VQMRBFZSX665ZRGRQDENEAWGZBXFGJ3MCA, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:21:55.933205363 (30m11.714138735s) [noise=true , address=79.127.220.101:4048 , piece_id=HN6H6ZIBWVWEPBUVSDKPZETXHOGED6VPSBXD4TNSL7FWBE6JMYCA, seg_pos={0 0}, upload=471]: Done uploading piece. err: <nil> Jul 26 22:22:51.089724129 (31m6.870657519s) [noise=true , address=104.36.82.4:4823 , piece_id=TH253XT6ANY3TQHCZPYHXXRXNEYTB5GFWOYAQKUPSOZK4RIQVBDQ, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:1JWhB5QxbZsTD863tEfkEN1CUbKBMQvMJ9ZaiMAwKvxxXyDkEq, address:104.36.82.4:4823): protocol: write tcp 159.223.143.39:40584->104.36.82.4:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.089835968 (31m6.870769352s) [noise=true , address=104.36.82.2:4751 , piece_id=YKHFYDOUVJKIIX3FYWAS23AMQP7HDLB4HWNPB2KUHWOAYCZOYTWA, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:104.36.82.2:4751): protocol: write tcp 159.223.143.39:52820->104.36.82.2:4751: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.089875703 (31m6.870809076s) [noise=true , address=104.36.82.10:4090 , piece_id=26Y24VLIT4PILDM7PQQG5ZGG3T6O6KRKECW3W3FLQFYYID3RVBIA, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:104.36.82.10:4090): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:22:51.089916710 (31m6.870850099s) [noise=true , address=104.36.82.13:4193 , piece_id=X4H7E5467UIBFY6YJUNHWCYJ6JT33OR6BPV3FHZTEHDJPUNST7OA, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:15XNLgptmXprcfkVMJsuzY2WDyHiZDpLetiBpmjXhb4c9q7yMd, address:104.36.82.13:4193): protocol: write tcp 159.223.143.39:57854->104.36.82.13:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.089972701 (31m6.870906074s) [noise=true , address=104.36.82.11:4147 , piece_id=SDXVXM5HBHOF4TD7FL5SXUSNMELHZX5KHPXKDXZOCX7QGJ4DMEEQ, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:12nNFCM4wDSvGMeqF8MF6csgKDUV6QyozoC5fZwUaj7MN1Woz4P, address:104.36.82.11:4147): protocol: rpc: dial tcp 104.36.82.11:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.089986736 (31m6.870920122s) [noise=true , address=104.36.82.19:4408 , piece_id=3G3ENNDUYJV2BPFQ7Q7P54LGRPZRWEY5SFIKNHKEE6XOXOJ4TBBA, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:12evzqLGVaxayKpKejqBXNLwqTKyKNKa94NZYJBUCdvyUo3vhGJ, address:104.36.82.19:4408): protocol: write tcp 159.223.143.39:37316->104.36.82.19:4408: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.090067672 (31m6.871001065s) [noise=true , address=104.36.82.34:4551 , piece_id=4IOL6UNVF3LW5G5JWLNJUEOVMMNKD4F7CBJJ5NXSD2BQYSXFPEJA, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:12vR5WRMGoYyzYDZzwELFw6kLG6vtKsUeFM9kUhgZDLGKB6vcXs, address:104.36.82.34:4551): protocol: rpc: dial tcp 104.36.82.34:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.090097438 (31m6.871030811s) [noise=true , address=104.36.82.41:4795 , piece_id=FUOCDY3HJEZHJMDLFJCIF72XWWXCBHVS2AMWCD5L4FRBMJ5S4QCQ, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:12ovwD5mScxxBf7RtRjAbBqRiqgAmTC8ajN3oBAXSaWrDQ1t9xk, address:104.36.82.41:4795): protocol: write tcp 159.223.143.39:35918->104.36.82.41:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.090142495 (31m6.87107586s) [noise=true , address=104.36.82.12:4164 , piece_id=SKNQD6JQQET7CTI6OU5FSA2WUECERJPMHKE34SHW5RPFANWML2PQ, seg_pos={0 0}, upload=471]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: rpc: dial tcp 104.36.82.12:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:22:51.090182806 (31m6.871116183s) [seg_pos={0 0}, upload=471]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:22:51.127000051 (31m6.907933421s) [upload=471]: Done uploading segments... Jul 26 22:22:51.127086170 (31m6.908019549s) [upload=471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.040562238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....................XX..XXX..X..XX..X..XX.X....X....X......] duration: 457.703373ms FAIL (2025-07-26 22:22:39 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.737426ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 555.760673ms FAIL (2025-07-26 22:22:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 555.788484ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..................X..X........XXX.XXX.X.XX..X..X......X.....] duration: 516.158254ms FAIL (2025-07-26 22:22:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 516.196568ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................X..X........XXX.XXX.X.XX..X..X......X....] duration: 511.578483ms FAIL (2025-07-26 22:22:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.607839ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......................XX..XXX..X..XX..X..XX.X....X....X.....] duration: 587.842018ms FAIL (2025-07-26 22:22:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 587.894097ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 642.918336ms FAIL (2025-07-26 22:21:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 642.94344ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 521.932669ms FAIL (2025-07-26 22:21:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 521.965326ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 1m0.066412599s FAIL (2025-07-26 22:21:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== oad segment. Jul 26 22:20:40.916534212 (28m56.697467601s) [noise=true , address=104.36.82.12:4155 , piece_id=E74ZU3DO5PIVWT2WRTRNTNWG7TK23ZKODT5V6JYU7W7AHWXYRI7Q, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.916570964 (28m56.697504387s) [noise=true , address=45.91.116.13:4187 , piece_id=2MAZJ3YW2EDAW3MKB2YMUSQ76UNEHXK24PTLXGUU7RI5HY5M2N5Q, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.917064939 (28m56.697998328s) [noise=true , address=79.127.219.47:4069 , piece_id=TNSFWZDTGDLFIEGG33CUHDZYRUPXOI4IXUVMPX2FIPFMVIHQQQKQ, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.917123624 (28m56.698057007s) [noise=true , address=104.36.82.44:4897 , piece_id=JVHMJRID3ABUCL6BSUPSHYC35SLT2EY4OQCDA3ULS2EUWUQGVXHQ, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.917510393 (28m56.698443779s) [noise=true , address=79.127.205.235:4691 , piece_id=T5RV3QXAGTQA37QPMNFVN732UW54GNHO3KJQYZLPHFMEAPNHAHIQ, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.917633410 (28m56.698566793s) [noise=true , address=79.127.205.245:4244 , piece_id=UEZK64QNEQO5MFHUB3LHKLWQ3I3ICTIIXLBWWP7JO6GUPS4DLBQA, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.917943650 (28m56.698877034s) [noise=true , address=79.127.201.215:4358 , piece_id=5YDLANQUQVF2K7RRJA7RJM5C3C4THNG3P3U52ZOB6ZLBP5SUEGSA, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.918050367 (28m56.698983751s) [noise=true , address=104.36.82.21:4507 , piece_id=45FVIHVVJLRNR5FK7VRURRZZXXPPBE5QNAVV22G5POP52GRNZHKA, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.918365786 (28m56.699299151s) [noise=true , address=104.36.82.43:4865 , piece_id=P2VJ4LDTQ4FK35UNPWOYOVUUM2VKTYQBQJP6KPKS7UOJCCLYHOBQ, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.918505859 (28m56.699439245s) [noise=true , address=104.36.82.42:4827 , piece_id=U57EEL3Z2NSM7GIVERWBUAP3IBXAEZ7ROCNANPOE3F6DBWI4CT6Q, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.918774975 (28m56.699708361s) [noise=true , address=45.91.116.9:4269 , piece_id=YVZOBQJUDL5RH2Q7ASHRSJQVI2UX7S5D2GQ2HYR45ZEIM7CK7SMQ, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.918931799 (28m56.699865184s) [noise=true , address=104.36.82.41:4791 , piece_id=5AIEGNKLYA7DWAZX4BGYPV3JZWLT3QMHFQQRRYLVYYOKZGQGXR5Q, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.919409343 (28m56.70034273s) [noise=true , address=64.49.13.26:4947 , piece_id=2ZMF7G2YAZLRQABS6JNU6HRD4XC6IZOMPDRKN4AJYNSBLLXN2GGA, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.919844026 (28m56.700777405s) [noise=true , address=45.91.116.10:4100 , piece_id=WYAEZ2ZBKVFKF3RXUFCQNP4WVWKIBGPVQXRJJG6MA2FOYA56FLXQ, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.920546060 (28m56.70147945s) [noise=true , address=104.36.82.35:4587 , piece_id=VYKCOUYP2JPO4XBUZL4LGL3JJ5ZJHJLOPBQZM4QHZTCHUQ6YTPWQ, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.920999774 (28m56.701933162s) [noise=true , address=104.36.82.40:4942 , piece_id=INLNCL2EOJJ7XCXQV2D3TBFE5MXH76FBZ2QSMV425WJTS2SYSMHA, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.921393132 (28m56.702326527s) [noise=true , address=77.90.31.13:4658 , piece_id=YPPCQ7FGIRQU7EWDTMMTDTUGXCSXJQA6KVFHZNGOSURSJS3CZ2FA, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.921893347 (28m56.702826736s) [noise=true , address=77.90.31.10:4531 , piece_id=JT7UNCP3SLMVUMLVLFQVDZMPIXGW42M7OJ3SNJIM5ELW7L6UHNYA, seg_pos={0 0}, upload=450]: Uploading piece... Jul 26 22:20:40.971970107 (28m56.752903488s) [noise=true , address=77.90.31.13:4658 , piece_id=YPPCQ7FGIRQU7EWDTMMTDTUGXCSXJQA6KVFHZNGOSURSJS3CZ2FA, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: storage node overloaded, request limit: 2000 Jul 26 22:20:41.021641688 (28m56.802575072s) [upload=450]: Next returned nil segment Jul 26 22:20:41.021663651 (28m56.802597037s) [upload=450]: Waiting for error group managing segments... Jul 26 22:20:41.116679795 (28m56.897613165s) [noise=true , address=77.90.31.10:4531 , piece_id=JT7UNCP3SLMVUMLVLFQVDZMPIXGW42M7OJ3SNJIM5ELW7L6UHNYA, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:20:41.305840694 (28m57.086774061s) [noise=true , address=79.127.219.47:4069 , piece_id=TNSFWZDTGDLFIEGG33CUHDZYRUPXOI4IXUVMPX2FIPFMVIHQQQKQ, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:20:41.510544637 (28m57.291477998s) [noise=true , address=79.127.205.235:4691 , piece_id=T5RV3QXAGTQA37QPMNFVN732UW54GNHO3KJQYZLPHFMEAPNHAHIQ, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:20:41.573772645 (28m57.354706027s) [noise=true , address=79.127.205.245:4244 , piece_id=UEZK64QNEQO5MFHUB3LHKLWQ3I3ICTIIXLBWWP7JO6GUPS4DLBQA, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:20:41.978973318 (28m57.759906698s) [noise=true , address=64.49.13.26:4947 , piece_id=2ZMF7G2YAZLRQABS6JNU6HRD4XC6IZOMPDRKN4AJYNSBLLXN2GGA, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:20:42.722060157 (28m58.502993537s) [noise=true , address=79.127.201.215:4358 , piece_id=5YDLANQUQVF2K7RRJA7RJM5C3C4THNG3P3U52ZOB6ZLBP5SUEGSA, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:20:43.216586384 (28m58.997519763s) [noise=true , address=45.91.116.9:4269 , piece_id=YVZOBQJUDL5RH2Q7ASHRSJQVI2UX7S5D2GQ2HYR45ZEIM7CK7SMQ, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:20:50.316596217 (29m6.097529596s) [noise=true , address=45.91.116.13:4187 , piece_id=2MAZJ3YW2EDAW3MKB2YMUSQ76UNEHXK24PTLXGUU7RI5HY5M2N5Q, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:21:11.727732259 (29m27.508665637s) [noise=true , address=45.91.116.10:4100 , piece_id=WYAEZ2ZBKVFKF3RXUFCQNP4WVWKIBGPVQXRJJG6MA2FOYA56FLXQ, seg_pos={0 0}, upload=450]: Done uploading piece. err: <nil> Jul 26 22:21:40.840711299 (29m56.621644677s) [noise=true , address=104.36.82.44:4897 , piece_id=JVHMJRID3ABUCL6BSUPSHYC35SLT2EY4OQCDA3ULS2EUWUQGVXHQ, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:104.36.82.44:4897): protocol: rpc: dial tcp 104.36.82.44:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.840803900 (29m56.621737284s) [noise=true , address=104.36.82.43:4865 , piece_id=P2VJ4LDTQ4FK35UNPWOYOVUUM2VKTYQBQJP6KPKS7UOJCCLYHOBQ, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:12DQ2GHBBNrbKsE6eo3YTsMkQGuekVYwrQQjBdQWtQWNgd35VPQ, address:104.36.82.43:4865): protocol: rpc: dial tcp 104.36.82.43:4865: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.840890063 (29m56.621823441s) [noise=true , address=104.36.82.42:4827 , piece_id=U57EEL3Z2NSM7GIVERWBUAP3IBXAEZ7ROCNANPOE3F6DBWI4CT6Q, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:1pqAbf2u3J689bYTQUaaxWdF4R241i7qYHraSWK14nV68pSUfC, address:104.36.82.42:4827): protocol: rpc: dial tcp 104.36.82.42:4827: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.841165317 (29m56.622098701s) [noise=true , address=104.36.82.21:4507 , piece_id=45FVIHVVJLRNR5FK7VRURRZZXXPPBE5QNAVV22G5POP52GRNZHKA, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: rpc: dial tcp 104.36.82.21:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.841237756 (29m56.622171134s) [noise=true , address=104.36.82.35:4587 , piece_id=VYKCOUYP2JPO4XBUZL4LGL3JJ5ZJHJLOPBQZM4QHZTCHUQ6YTPWQ, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY, address:104.36.82.35:4587): protocol: rpc: dial tcp 104.36.82.35:4587: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.841313429 (29m56.622246806s) [noise=true , address=104.36.82.12:4155 , piece_id=E74ZU3DO5PIVWT2WRTRNTNWG7TK23ZKODT5V6JYU7W7AHWXYRI7Q, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:104.36.82.12:4155): protocol: rpc: dial tcp 104.36.82.12:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.841376910 (29m56.622310288s) [noise=true , address=104.36.82.41:4791 , piece_id=5AIEGNKLYA7DWAZX4BGYPV3JZWLT3QMHFQQRRYLVYYOKZGQGXR5Q, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:12LpFW1mfotWPqgQxyBSimChkttujx21L6rU2GRSwvqbZ7CVeLn, address:104.36.82.41:4791): protocol: rpc: dial tcp 104.36.82.41:4791: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.841454993 (29m56.62238837s) [noise=true , address=104.36.82.40:4942 , piece_id=INLNCL2EOJJ7XCXQV2D3TBFE5MXH76FBZ2QSMV425WJTS2SYSMHA, seg_pos={0 0}, upload=450]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:104.36.82.40:4942): protocol: write tcp 159.223.143.39:42564->104.36.82.40:4942: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:21:40.841474477 (29m56.62240787s) [seg_pos={0 0}, upload=450]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:21:40.900794912 (29m56.681728291s) [upload=450]: Done uploading segments... Jul 26 22:21:40.900829171 (29m56.681762555s) [upload=450]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.066508586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 14.900055419s FAIL (2025-07-26 22:21:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.900087912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 40.773827277s FAIL (2025-07-26 22:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.773862811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................X..X........XXX.XXX.X.XX..X..X......X...] duration: 472.802858ms FAIL (2025-07-26 22:21:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.836773ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................XX..XXX..X..XX..X..XX.X....X....X....] duration: 410.554499ms FAIL (2025-07-26 22:20:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.606838ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....................X..X........XXX.XXX.X.XX..X..X......X..] duration: 508.800386ms FAIL (2025-07-26 22:20:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 508.845979ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 572.886484ms FAIL (2025-07-26 22:20:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 572.986166ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 1m0.040803194s FAIL (2025-07-26 22:20:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ]: Uploading piece... Jul 26 22:19:27.235986107 (27m43.016919488s) [noise=true , address=104.36.82.41:4778 , piece_id=FLGNR7WX7GZNL5XLCCF233IDACX2ZJTCLHJDKZGR5EBE5DT5TPMA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.236061262 (27m43.016994639s) [noise=true , address=79.127.205.234:4617 , piece_id=RY75UR24SEU7POCQC4MMWJMUZVL7LVI3K4AN6OU6OD3FYBIM777A, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.237084512 (27m43.01801791s) [noise=true , address=104.36.82.55:4199 , piece_id=SCXPHH5MFWDKAS7YUQIRZWAI5TNMO4JOZYNPTYDEHEHTYJRWX2CA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.237388855 (27m43.018322242s) [noise=true , address=104.36.82.2:4744 , piece_id=E2DNQWIRGNAYEPLIRRSZZT4JHZSR3TSICEXG4VPFTTBUWGXY4SQQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.237849474 (27m43.018782864s) [noise=true , address=64.49.13.170:4718 , piece_id=ARVQ5VTCTT6MA5NRY4CSLEG2MZ3UAVFZYL3YSUR2WQUID5VXXPPA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238228951 (27m43.019162338s) [noise=true , address=77.90.31.11:4560 , piece_id=L7IJ2ZJRLTELRD4IBCNCOI5G3PTEBP7QEH767LB7HEJ6D7673YBQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238618860 (27m43.019552239s) [noise=true , address=104.36.82.52:4109 , piece_id=DFJQPIHGD4XJH6PEEJWBMRM6AJWACYLPXJIDGXZ6PYIZKHKRVN5Q, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238933310 (27m43.019866695s) [noise=true , address=45.91.116.14:4222 , piece_id=GFC3ZCOT5T4S7M6KOY2MAUXCCIMFPKCVUC6D5K6M5LGGJ6M4UK6Q, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.238998088 (27m43.019931465s) [noise=true , address=104.36.82.56:4261 , piece_id=QPZDHSVKAZFTWZUXKWGGJ5HIESZYJ4UCA6SAEOZ4RASY45LL445Q, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.246839994 (27m43.027773382s) [noise=true , address=77.90.31.16:4754 , piece_id=EEZ2M3MPSJ54RPOXSNZ2LHH7J36BLVBZZLPMHHNCEZZ4XU3DE4PA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.247343875 (27m43.028277261s) [noise=true , address=64.49.13.155:4923 , piece_id=SBBVP4SU7SSWCMGEDTYSU53LRS3SOLBRSJHCDHPDAIPL4WDQGIIQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.262390738 (27m43.043324127s) [noise=true , address=104.36.82.6:4908 , piece_id=BX6KPDECSK72CMBOBZ4434NGJ76TDGJVSCTDFE3WKIQTP4RFRAFQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.265715820 (27m43.04664921s) [noise=true , address=104.36.82.50:4032 , piece_id=QIWESO6HLFAZ3LWATNRYPSCDAFN65IBJXW6ZHBPR3F4KWLHY7UNQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.269101909 (27m43.050035297s) [noise=true , address=104.36.82.47:4519 , piece_id=RAU7UWKJKPGRKHETYIZ6CWQV4KSJPULVXDKGXY24F2UIR2I6CMLA, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.271831206 (27m43.052764594s) [noise=true , address=104.36.82.21:4476 , piece_id=SWYERXQJN4ATA4IYDWZWAV4SILBJDC2VZFQDHBD5W33HTOEYQVAQ, seg_pos={0 0}, upload=434]: Uploading piece... Jul 26 22:19:27.324992170 (27m43.105925549s) [upload=434]: Next returned nil segment Jul 26 22:19:27.325016561 (27m43.10594993s) [upload=434]: Waiting for error group managing segments... Jul 26 22:19:27.378650484 (27m43.159583847s) [noise=true , address=77.90.31.16:4754 , piece_id=EEZ2M3MPSJ54RPOXSNZ2LHH7J36BLVBZZLPMHHNCEZZ4XU3DE4PA, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:27.378759348 (27m43.159692731s) [noise=true , address=77.90.31.11:4560 , piece_id=L7IJ2ZJRLTELRD4IBCNCOI5G3PTEBP7QEH767LB7HEJ6D7673YBQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:27.935102012 (27m43.716035378s) [noise=true , address=79.127.219.48:4090 , piece_id=YSMZH7YAAXL2G7BZEYNWZI4HDRFYQAL2RHKIKFQ42HAGNX6GWJVQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:28.107941349 (27m43.888874731s) [noise=true , address=64.49.13.155:4923 , piece_id=SBBVP4SU7SSWCMGEDTYSU53LRS3SOLBRSJHCDHPDAIPL4WDQGIIQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:28.134829073 (27m43.915762465s) [noise=true , address=79.127.205.234:4617 , piece_id=RY75UR24SEU7POCQC4MMWJMUZVL7LVI3K4AN6OU6OD3FYBIM777A, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:29.426104769 (27m45.207038133s) [noise=true , address=64.49.13.170:4718 , piece_id=ARVQ5VTCTT6MA5NRY4CSLEG2MZ3UAVFZYL3YSUR2WQUID5VXXPPA, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:19:33.820352231 (27m49.601285608s) [noise=true , address=45.91.116.14:4222 , piece_id=GFC3ZCOT5T4S7M6KOY2MAUXCCIMFPKCVUC6D5K6M5LGGJ6M4UK6Q, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:20:18.653274869 (28m34.434208255s) [noise=true , address=104.36.82.6:4908 , piece_id=BX6KPDECSK72CMBOBZ4434NGJ76TDGJVSCTDFE3WKIQTP4RFRAFQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: <nil> Jul 26 22:20:27.169426341 (28m42.950359722s) [noise=true , address=104.36.82.47:4519 , piece_id=RAU7UWKJKPGRKHETYIZ6CWQV4KSJPULVXDKGXY24F2UIR2I6CMLA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: read tcp 159.223.143.39:37040->104.36.82.47:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.169630094 (28m42.950563473s) [noise=true , address=104.36.82.21:4476 , piece_id=SWYERXQJN4ATA4IYDWZWAV4SILBJDC2VZFQDHBD5W33HTOEYQVAQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12NKwRvjiEpY7DrbqFGaDHQ8no5YV4cUALnpsvQ8icUPoLvMYen, address:104.36.82.21:4476): protocol: write tcp 159.223.143.39:55340->104.36.82.21:4476: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.169768451 (28m42.950701836s) [noise=true , address=104.36.82.56:4261 , piece_id=QPZDHSVKAZFTWZUXKWGGJ5HIESZYJ4UCA6SAEOZ4RASY45LL445Q, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: read tcp 159.223.143.39:40862->104.36.82.56:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.169872049 (28m42.950805428s) [noise=true , address=104.36.82.50:4032 , piece_id=QIWESO6HLFAZ3LWATNRYPSCDAFN65IBJXW6ZHBPR3F4KWLHY7UNQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1zj1d7Yzk4Z5Wda3YF6UgR5oSEhABcf6HCWwmHWXaX1CbKYJdQ, address:104.36.82.50:4032): protocol: read tcp 159.223.143.39:58490->104.36.82.50:4032: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.170988366 (28m42.951921756s) [noise=true , address=104.36.82.53:4153 , piece_id=QZ7HXROJ2XTQPFL6I4WHHIX7EGM22MZDUSW4YSMNQRQXT77IN3GA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 159.223.143.39:43288->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171211780 (28m42.952145168s) [noise=true , address=104.36.82.46:4490 , piece_id=HI4BCBNOHZYRSFJIJVKA64EYS3UFWSANSK47D3NRALBSYHVDJVPA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: read tcp 159.223.143.39:39554->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171280229 (28m42.952213609s) [noise=true , address=104.36.82.41:4778 , piece_id=FLGNR7WX7GZNL5XLCCF233IDACX2ZJTCLHJDKZGR5EBE5DT5TPMA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: write tcp 159.223.143.39:38720->104.36.82.41:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171616785 (28m42.952550169s) [noise=true , address=104.36.82.55:4199 , piece_id=SCXPHH5MFWDKAS7YUQIRZWAI5TNMO4JOZYNPTYDEHEHTYJRWX2CA, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: write tcp 159.223.143.39:42032->104.36.82.55:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171682575 (28m42.952615965s) [noise=true , address=104.36.82.2:4744 , piece_id=E2DNQWIRGNAYEPLIRRSZZT4JHZSR3TSICEXG4VPFTTBUWGXY4SQQ, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:104.36.82.2:4744): protocol: write tcp 159.223.143.39:43896->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:20:27.171956497 (28m42.952889883s) [noise=true , address=104.36.82.52:4109 , piece_id=DFJQPIHGD4XJH6PEEJWBMRM6AJWACYLPXJIDGXZ6PYIZKHKRVN5Q, seg_pos={0 0}, upload=434]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: write tcp 159.223.143.39:55594->104.36.82.52:4109: use of closed network connection; conn closed: op Jul 26 22:20:27.171983490 (28m42.952916881s) [seg_pos={0 0}, upload=434]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:20:27.196484578 (28m42.977417944s) [upload=434]: Done uploading segments... Jul 26 22:20:27.196520272 (28m42.977453652s) [upload=434]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.040837011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [......................X..X........XXX.XXX.X.XX..X..X......X.] duration: 630.902017ms FAIL (2025-07-26 22:20:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 630.927609ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................X..X........XXX.XXX.X.XX..X..X......X] duration: 30.085567921s FAIL (2025-07-26 22:20:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.085591667s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........................XX..XXX..X..XX..X..XX.X....X....X...] duration: 454.357217ms FAIL (2025-07-26 22:19:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.400063ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 511.32734ms FAIL (2025-07-26 22:19:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 511.360827ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.173404495s FAIL (2025-07-26 22:19:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.173435831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 16.069552051s FAIL (2025-07-26 22:19:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.658933536 (26m17.439866923s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659377297 (26m17.440310683s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659425641 (26m17.440359043s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659891300 (26m17.440824687s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660027462 (26m17.44096085s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660516844 (26m17.441450237s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660882579 (26m17.441815965s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660990717 (26m17.441924107s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661480854 (26m17.442414245s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661498728 (26m17.442432116s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661930324 (26m17.44286375s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662077993 (26m17.443011373s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662543009 (26m17.443476396s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.664341220 (26m17.445274608s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.772808761 (26m17.55374215s) [upload=408]: Next returned nil segment Jul 26 22:18:01.772831167 (26m17.553764552s) [upload=408]: Waiting for error group managing segments... Jul 26 22:18:01.799611104 (26m17.580544483s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:01.819792373 (26m17.600725758s) [noise=true , address=77.90.31.13:4658 , piece_id=ATJN5WMNODMLTCPGDPHFNOH7BM7PSH75F5GXCC7EXNBFWZWA4GEQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.282782349 (26m18.063715732s) [noise=true , address=79.127.205.235:4688 , piece_id=WGW73TAYOCEWTHBPJTNOF6NDSKHG7AZZ2W6WOWU4KASYV4MM5W5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.376301883 (26m18.157235262s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:04.945231567 (26m20.72616493s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:05.987660817 (26m21.768594198s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:07.487167062 (26m23.268100445s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:19:01.591229652 (27m17.37216303s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: write tcp 159.223.143.39:40798->104.36.82.14:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591410483 (27m17.372343861s) [noise=true , address=104.36.82.50:4041 , piece_id=MNTR6WWQAF7YTJWXT3AJKLCEGJE3REYN5H2SBNCGLMZPKEC5XYSA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12HSwcUKtymKKFJYd9acgwgQym2owswGXG8h61HAWdP7uooFCfk, address:104.36.82.50:4041): protocol: write tcp 159.223.143.39:44652->104.36.82.50:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591467344 (27m17.372400722s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:104.36.82.17:4362): protocol: write tcp 159.223.143.39:42742->104.36.82.17:4362: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591745171 (27m17.37267855s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591824050 (27m17.372757428s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 159.223.143.39:39950->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591898010 (27m17.372831391s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59, address:104.36.82.41:4788): protocol: write tcp 159.223.143.39:33814->104.36.82.41:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591966690 (27m17.372900068s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: write tcp 159.223.143.39:35654->104.36.82.53:4143: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592038412 (27m17.37297179s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 159.223.143.39:50510->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592113478 (27m17.37304686s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: write tcp 159.223.143.39:58296->104.36.82.55:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592380636 (27m17.37331402s) [noise=true , address=104.36.82.47:4528 , piece_id=RUUKRJW5LIZSO3MSDQYXN6HIWNGIHCBZZTIHJGH2IU75Y43MMTOQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: read tcp 159.223.143.39:46510->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592666167 (27m17.373599555s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:01.592698834 (27m17.373632223s) [seg_pos={0 0}, upload=408]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:19:01.617673696 (27m17.398607076s) [upload=408]: Done uploading segments... Jul 26 22:19:01.617726168 (27m17.398659549s) [upload=408]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 16.069667095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........................XX..XXX..X..XX..X..XX.X....X....X..] duration: 624.608196ms FAIL (2025-07-26 22:19:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 624.689159ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 648.888329ms FAIL (2025-07-26 22:19:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 648.921931ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031094754s FAIL (2025-07-26 22:19:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.658933536 (26m17.439866923s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659377297 (26m17.440310683s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659425641 (26m17.440359043s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.659891300 (26m17.440824687s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660027462 (26m17.44096085s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660516844 (26m17.441450237s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660882579 (26m17.441815965s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.660990717 (26m17.441924107s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661480854 (26m17.442414245s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661498728 (26m17.442432116s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.661930324 (26m17.44286375s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662077993 (26m17.443011373s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.662543009 (26m17.443476396s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.664341220 (26m17.445274608s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Uploading piece... Jul 26 22:18:01.772808761 (26m17.55374215s) [upload=408]: Next returned nil segment Jul 26 22:18:01.772831167 (26m17.553764552s) [upload=408]: Waiting for error group managing segments... Jul 26 22:18:01.799611104 (26m17.580544483s) [noise=true , address=77.90.31.21:4882 , piece_id=SDVRZFULUPSAOUB4KJOPON67SY5DAVD53ZR3NHWRII767KVPURHQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:01.819792373 (26m17.600725758s) [noise=true , address=77.90.31.13:4658 , piece_id=ATJN5WMNODMLTCPGDPHFNOH7BM7PSH75F5GXCC7EXNBFWZWA4GEQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.282782349 (26m18.063715732s) [noise=true , address=79.127.205.235:4688 , piece_id=WGW73TAYOCEWTHBPJTNOF6NDSKHG7AZZ2W6WOWU4KASYV4MM5W5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:02.376301883 (26m18.157235262s) [noise=true , address=79.127.205.233:4642 , piece_id=3DOJD5AM3Q4OT5UUYQNHHYTI5VUB7ZPRBMYFQ4PC2FHV7CM2QRNQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:04.945231567 (26m20.72616493s) [noise=true , address=64.49.13.138:4482 , piece_id=GRDLRXVTXZZS5XVV6SHQUB6UGUK6CPHB7YYNSPCP7CSMCWYDUOKQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:05.987660817 (26m21.768594198s) [noise=true , address=156.146.43.225:4449 , piece_id=TBABU6V64MD4QLQ473T65KXONG6PBMWONURJLRVTFGPWUR6R2KPA, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:18:07.487167062 (26m23.268100445s) [noise=true , address=79.127.201.215:4352 , piece_id=SDPANTM6JWR66IA3NSHZDXQHKEA6ZEUBKIGXBED3BWMBCWSI6WDQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: <nil> Jul 26 22:19:01.591229652 (27m17.37216303s) [noise=true , address=104.36.82.14:4256 , piece_id=P4ZAQ7RT3QUEGNGUXCJX4LSW3YREEECQT6P2NBFP4QQADQVE4XXQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: write tcp 159.223.143.39:40798->104.36.82.14:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591410483 (27m17.372343861s) [noise=true , address=104.36.82.50:4041 , piece_id=MNTR6WWQAF7YTJWXT3AJKLCEGJE3REYN5H2SBNCGLMZPKEC5XYSA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12HSwcUKtymKKFJYd9acgwgQym2owswGXG8h61HAWdP7uooFCfk, address:104.36.82.50:4041): protocol: write tcp 159.223.143.39:44652->104.36.82.50:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591467344 (27m17.372400722s) [noise=true , address=104.36.82.17:4362 , piece_id=27RJ4NKLFJBFDXWQWE3RC3KRAVSREHMRPHKC45ZYDSYJA2V75V5Q, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:123zeRoJ5g7juzsV7qQeNacC5BB7jZpfDBnznFhySGWfyPGJ4oa, address:104.36.82.17:4362): protocol: write tcp 159.223.143.39:42742->104.36.82.17:4362: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591745171 (27m17.37267855s) [noise=true , address=104.36.82.13:4212 , piece_id=Y24ZROTUDCEIUOKOCIXGO2BV4YTTWOLY3OAIQNQHA4ITYJZTRSGA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591824050 (27m17.372757428s) [noise=true , address=104.36.82.52:4103 , piece_id=WIL3HNDLU7EQUBH4WCBISGED4BHCZ3LUF7JM6QPB3NBDEKZQ67XA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 159.223.143.39:39950->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591898010 (27m17.372831391s) [noise=true , address=104.36.82.41:4788 , piece_id=3K4ULBFCPSV4RQTVR574UHDPJVLEODJPTJPN5LUXUHDBWI5DQB3A, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59, address:104.36.82.41:4788): protocol: write tcp 159.223.143.39:33814->104.36.82.41:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.591966690 (27m17.372900068s) [noise=true , address=104.36.82.53:4143 , piece_id=WW7JWTTAD3PS2MRGOBZLJ4YBOWQP6DXTU2QVWDGZVW2BP3UIHJZQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: write tcp 159.223.143.39:35654->104.36.82.53:4143: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592038412 (27m17.37297179s) [noise=true , address=104.36.82.44:4896 , piece_id=2QPYNLBB4K2AX7HL73MTCIQOQWJHMBVU5IMLEELFAYCXQAS6IOEA, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 159.223.143.39:50510->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592113478 (27m17.37304686s) [noise=true , address=104.36.82.55:4211 , piece_id=HEACUCU4BUSJOVTF7U3Q4SAFE4ZFLE5S2LWTAMAP4GEGIFUIMCRQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: write tcp 159.223.143.39:58296->104.36.82.55:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592380636 (27m17.37331402s) [noise=true , address=104.36.82.47:4528 , piece_id=RUUKRJW5LIZSO3MSDQYXN6HIWNGIHCBZZTIHJGH2IU75Y43MMTOQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: read tcp 159.223.143.39:46510->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:19:01.592666167 (27m17.373599555s) [noise=true , address=104.36.82.46:4512 , piece_id=SRWIA33MM3MQ7NPI4VIC2OQ6F5UZ256VKWWZZQZ5HJLD3MTUKLPQ, seg_pos={0 0}, upload=408]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:19:01.592698834 (27m17.373632223s) [seg_pos={0 0}, upload=408]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:19:01.617673696 (27m17.398607076s) [upload=408]: Done uploading segments... Jul 26 22:19:01.617726168 (27m17.398659549s) [upload=408]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.031123277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................X..X........XXX.XXX.X.XX..X..X......] duration: 438.589903ms FAIL (2025-07-26 22:18:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 438.620741ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 11.204126513s FAIL (2025-07-26 22:18:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.204160093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................XX..XXX..X..XX..X..XX.X....X....X.] duration: 644.954476ms FAIL (2025-07-26 22:18:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 644.990413ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 575.763603ms FAIL (2025-07-26 22:18:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 575.787183ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................X..X........XXX.XXX.X.XX..X..X.....] duration: 542.317189ms FAIL (2025-07-26 22:18:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 542.344242ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.159094475s FAIL (2025-07-26 22:18:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.159115677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026301158s FAIL (2025-07-26 22:18:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 7N75M3XVA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.607834416 (25m17.388767809s) [noise=true , address=104.36.82.46:4511 , piece_id=OT66I7RL4BT44W26OYKS2DASSV3LXWE6IOS63SQ4ITD7JKEWV2IQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.608368825 (25m17.389302212s) [noise=true , address=104.36.82.20:4463 , piece_id=3LDMRO5O3F67KMXHUSLB2EM7P4OJMZSSQ32VZE32JINXCW27B3SQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.608765398 (25m17.38969878s) [noise=true , address=64.49.13.138:4482 , piece_id=RKK4PJBB24EFLE34Z3DACTF257T63JQARFS236ABEZ2ARBQ4E3WA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.609129021 (25m17.390062408s) [noise=true , address=104.36.82.21:4478 , piece_id=PZPSO6GX4CCMH6PXGV4OKCCAIRO2MZHSAEA4WO2JBOK334PKNXQQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.609503138 (25m17.390436525s) [noise=true , address=104.36.82.52:4114 , piece_id=7WXWHU57GSV54K2XCVUIRB6ZQJGIQV66A4MPCGWCDXWGZW27W2EA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.609856609 (25m17.390789997s) [noise=true , address=79.127.219.48:4090 , piece_id=24D7BBOVJWZBQECMHRA32MN5KMQMDF7HTQUPAOF7D6R5DGZD4QFQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.610205593 (25m17.391138979s) [noise=true , address=77.90.31.19:4875 , piece_id=YR5FX3GGFVCNS26Q7Y4XKWEHMU3RM2A4UVAEKACROJTBGZFJUXOA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.610610170 (25m17.39154355s) [noise=true , address=79.127.205.234:4610 , piece_id=SR7LZ4IUYRWBN6VSIF7W6KPQ7C2OU33JAZDPOT3STLPLZOBG6JHQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.611006218 (25m17.391939631s) [noise=true , address=79.127.201.215:4347 , piece_id=IAEEJE4VICZX7F3TIECDS5W3PHQ2V3KYWH4K42O7XL333AG4XVOA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.611425825 (25m17.39235921s) [noise=true , address=79.127.219.38:4007 , piece_id=CM26N5VO3VUP4XJTYWEGQWT3NLSXQFXXJFLNJUBDXVZVXUHNE65Q, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.611872771 (25m17.392806157s) [noise=true , address=104.36.82.47:4542 , piece_id=4LDMBNARMIDGDTZYSZ26DBUINO4KL4ZFEZ5BUNSMSFRI5IMKWYIA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.612281272 (25m17.39321466s) [noise=true , address=104.36.82.42:4807 , piece_id=PIAHLL22ZEXBV2LQTFMHSYSH57UZQQ3HZB3KT6AEOEZ5BWD6KEQQ, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.612682966 (25m17.393616345s) [noise=true , address=104.36.82.18:4382 , piece_id=S5MHPZ2MD2GIUWTGQLKEGO6KRA6A5UFKHBNLPTFZ7Q2KQJ7EG63Q, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.613057067 (25m17.393990444s) [noise=true , address=104.36.82.56:4233 , piece_id=Y3T2XHBZZ7E36QHCBPYZ2J7USGNXDJPBXYMULWBT7S326OHYQEAA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.613471934 (25m17.394405306s) [noise=true , address=104.36.82.53:4130 , piece_id=HKKR4UI5LX6VQT6KJJZ7EFILRC2YZX7VES3CJ3S7WMCX4KQBVHCA, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.613868892 (25m17.394802278s) [noise=true , address=104.36.82.55:4201 , piece_id=V4CJF6J7W76DQ7CPA52RPQDZ4LNC54G5QPMYDN4HCK2U2OMV5D6Q, seg_pos={0 0}, upload=392]: Uploading piece... Jul 26 22:17:01.714531762 (25m17.495465156s) [upload=392]: Next returned nil segment Jul 26 22:17:01.714556162 (25m17.495489552s) [upload=392]: Waiting for error group managing segments... Jul 26 22:17:01.730358742 (25m17.511292111s) [noise=true , address=79.127.219.38:4007 , piece_id=CM26N5VO3VUP4XJTYWEGQWT3NLSXQFXXJFLNJUBDXVZVXUHNE65Q, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:01.756445212 (25m17.537378598s) [noise=true , address=77.90.31.19:4875 , piece_id=YR5FX3GGFVCNS26Q7Y4XKWEHMU3RM2A4UVAEKACROJTBGZFJUXOA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.297674681 (25m18.07860805s) [noise=true , address=79.127.205.234:4610 , piece_id=SR7LZ4IUYRWBN6VSIF7W6KPQ7C2OU33JAZDPOT3STLPLZOBG6JHQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.315687352 (25m18.096620738s) [noise=true , address=156.146.43.225:4448 , piece_id=UV57RX32D2SEDOFU6MQP4WNNMJJGTL2HG3L7NSSBUQCCSSOWORZQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:17:02.430379841 (25m18.211313219s) [noise=true , address=79.127.219.48:4090 , piece_id=24D7BBOVJWZBQECMHRA32MN5KMQMDF7HTQUPAOF7D6R5DGZD4QFQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.503180683 (25m18.284114058s) [noise=true , address=79.127.205.245:4247 , piece_id=G6C2GIU7MO7POFDYT4GDYXFXLQQCXXGU5FAU2E74VYX7N75M3XVA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:02.854260496 (25m18.635193885s) [noise=true , address=64.49.13.138:4482 , piece_id=RKK4PJBB24EFLE34Z3DACTF257T63JQARFS236ABEZ2ARBQ4E3WA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:17:03.254464117 (25m19.035397498s) [noise=true , address=79.127.201.215:4347 , piece_id=IAEEJE4VICZX7F3TIECDS5W3PHQ2V3KYWH4K42O7XL333AG4XVOA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Jul 26 22:18:01.502194410 (26m17.283127794s) [noise=true , address=104.36.82.20:4463 , piece_id=3LDMRO5O3F67KMXHUSLB2EM7P4OJMZSSQ32VZE32JINXCW27B3SQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: rpc: dial tcp 104.36.82.20:4463: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502285807 (26m17.283219187s) [noise=true , address=104.36.82.21:4478 , piece_id=PZPSO6GX4CCMH6PXGV4OKCCAIRO2MZHSAEA4WO2JBOK334PKNXQQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:17kFjjJyvHApPUxUHJ6HeMEBCoo6xTbr8QizXjBEHwWj3mZUha, address:104.36.82.21:4478): protocol: rpc: dial tcp 104.36.82.21:4478: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502347090 (26m17.283280466s) [noise=true , address=104.36.82.52:4114 , piece_id=7WXWHU57GSV54K2XCVUIRB6ZQJGIQV66A4MPCGWCDXWGZW27W2EA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: rpc: dial tcp 104.36.82.52:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502411717 (26m17.283345094s) [noise=true , address=104.36.82.42:4807 , piece_id=PIAHLL22ZEXBV2LQTFMHSYSH57UZQQ3HZB3KT6AEOEZ5BWD6KEQQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1iTiX3X69T9rKBxe1MLwpJEJPZyXVATQBP9cDEnQe2riXMGZeZ, address:104.36.82.42:4807): protocol: rpc: dial tcp 104.36.82.42:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502456931 (26m17.283390309s) [noise=true , address=104.36.82.56:4233 , piece_id=Y3T2XHBZZ7E36QHCBPYZ2J7USGNXDJPBXYMULWBT7S326OHYQEAA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502506234 (26m17.283439611s) [noise=true , address=104.36.82.47:4542 , piece_id=4LDMBNARMIDGDTZYSZ26DBUINO4KL4ZFEZ5BUNSMSFRI5IMKWYIA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502550452 (26m17.283483828s) [noise=true , address=104.36.82.55:4201 , piece_id=V4CJF6J7W76DQ7CPA52RPQDZ4LNC54G5QPMYDN4HCK2U2OMV5D6Q, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502595223 (26m17.2835286s) [noise=true , address=104.36.82.18:4382 , piece_id=S5MHPZ2MD2GIUWTGQLKEGO6KRA6A5UFKHBNLPTFZ7Q2KQJ7EG63Q, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12S3SoJPMMyHpqDgG5Ry1KErPGHXugcTpikGNPjpwC7qq5sH9ho, address:104.36.82.18:4382): protocol: rpc: dial tcp 104.36.82.18:4382: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.502691161 (26m17.28362454s) [noise=true , address=104.36.82.46:4511 , piece_id=OT66I7RL4BT44W26OYKS2DASSV3LXWE6IOS63SQ4ITD7JKEWV2IQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: rpc: dial tcp 104.36.82.46:4511: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:18:01.504258618 (26m17.285191985s) [noise=true , address=104.36.82.53:4130 , piece_id=HKKR4UI5LX6VQT6KJJZ7EFILRC2YZX7VES3CJ3S7WMCX4KQBVHCA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12N8MGboFyEV381rkrBva3G3advz3QQEY2HCHxwqzKX2X5zxNvb, address:104.36.82.53:4130): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:18:01.504316278 (26m17.285249663s) [seg_pos={0 0}, upload=392]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:18:01.526127706 (26m17.307061086s) [upload=392]: Done uploading segments... Jul 26 22:18:01.526182580 (26m17.307115959s) [upload=392]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.026321347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 460.880534ms FAIL (2025-07-26 22:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.902254ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........................XX..XXX..X..XX..X..XX.X....X....X] duration: 30.031937519s FAIL (2025-07-26 22:17:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031963496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........................X..X........XXX.XXX.X.XX..X..X....] duration: 2.448114829s FAIL (2025-07-26 22:17:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.448143588s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.88296956s FAIL (2025-07-26 22:17:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.883081671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033562229s FAIL (2025-07-26 22:17:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== ) [noise=true , address=104.36.82.53:4135 , piece_id=AMR4IJL6GIO2LWYKZO5NJ4IS3R45DYGLTDSNCKRI3JLP3YZV2ECA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.473745003 (24m17.25467839s) [noise=true , address=104.36.82.46:4514 , piece_id=LGVYOFWWLES44276UBMMALIL36Z3FJ3LOPTX5U2OGCHOG4E3B6DQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.473811231 (24m17.254744614s) [noise=true , address=104.36.82.56:4234 , piece_id=VNGTB2FLRW44V5ZE2TITOPNJYCRTQTGKPQVMUAKTTEZKCAWNZG2A, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.474250703 (24m17.255184091s) [noise=true , address=79.127.205.235:4682 , piece_id=EHTGCI7JSM642AATX23BVUFONM4APDHVIYMQAUNYLQT5MYGUZWIA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.474710580 (24m17.255643971s) [noise=true , address=64.49.13.138:4482 , piece_id=LTL6VLIEMLE7HIZXW2JRDXZFJ3VKCRLUECYKJIAT6KDRO6XTLABQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.474935918 (24m17.255869323s) [noise=true , address=79.127.201.215:4347 , piece_id=ODMZNTUSZCQHBR7SFU7PX3I2FE66WOBKOOKCMZ637QVVGYUKDG5Q, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.475565929 (24m17.256499317s) [noise=true , address=104.36.82.55:4203 , piece_id=QSGEUM7G42AUEG5JQ6CBAZQCNYNKEOL6TOX7CB5YF2ID4H6I6ICA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.475673696 (24m17.256607084s) [noise=true , address=104.36.82.11:4138 , piece_id=6HG5FAUP4TNT3DF2E2MOAEDD22LKWM7XQMJ6JUJOFAPEPAQMQKMA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.476048275 (24m17.256981665s) [noise=true , address=104.36.82.13:4219 , piece_id=SUUXCBWMZ3ZIVQCNBLZU33XZSTKYBDJPNYF7ATZ2ABTFBYPLZEUA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.476220112 (24m17.257153506s) [noise=true , address=79.127.219.38:4004 , piece_id=POKGXH5KKLJCSZYC5FZAX3PNM5KCQMINRP5XDPEWBYALQHCDP6NQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.476972597 (24m17.257905985s) [noise=true , address=104.36.82.17:4355 , piece_id=YATOCGA6ZHIJ4DMBBJIUWGYIJP2UZSZ76RXYGAZQT2UAEBXITVIA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.477444413 (24m17.258377803s) [noise=true , address=104.36.82.10:4080 , piece_id=4XZAMSGU4QDZSSQOFC5D6EDRLXUWTEPPMUDOUB5INXOIY5CO6G5Q, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.478256019 (24m17.259189397s) [noise=true , address=104.36.82.52:4115 , piece_id=TQSWEH44YYCIQA2TXJ7VECTWQPW3NKZ4ELII5RFVXATKUEQJLTRQ, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.478611307 (24m17.259544693s) [noise=true , address=77.90.31.18:4821 , piece_id=WPM7KD6YIQG5SEHWZCNWJAE6KJDF7HXOKT7Q4XNULD3DPAY6QZVA, seg_pos={0 0}, upload=378]: Uploading piece... Jul 26 22:16:01.589099580 (24m17.370032958s) [upload=378]: Next returned nil segment Jul 26 22:16:01.589117146 (24m17.370050524s) [upload=378]: Waiting for error group managing segments... Jul 26 22:16:01.701337336 (24m17.482270701s) [noise=true , address=77.90.31.18:4821 , piece_id=WPM7KD6YIQG5SEHWZCNWJAE6KJDF7HXOKT7Q4XNULD3DPAY6QZVA, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.087132899 (24m17.868066268s) [noise=true , address=79.127.205.233:4640 , piece_id=HLRZZNUM7SB5XIYZENTWDZ5TWX7DB5MFXOHPJCSFIGZUWX6APGHQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.115884906 (24m17.896818269s) [noise=true , address=79.127.219.38:4004 , piece_id=POKGXH5KKLJCSZYC5FZAX3PNM5KCQMINRP5XDPEWBYALQHCDP6NQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.132652514 (24m17.913585894s) [noise=true , address=79.127.205.235:4682 , piece_id=EHTGCI7JSM642AATX23BVUFONM4APDHVIYMQAUNYLQT5MYGUZWIA, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:02.154969447 (24m17.935902829s) [noise=true , address=64.49.13.138:4482 , piece_id=LTL6VLIEMLE7HIZXW2JRDXZFJ3VKCRLUECYKJIAT6KDRO6XTLABQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:16:08.512970892 (24m24.293904255s) [noise=true , address=79.127.201.215:4347 , piece_id=ODMZNTUSZCQHBR7SFU7PX3I2FE66WOBKOOKCMZ637QVVGYUKDG5Q, seg_pos={0 0}, upload=378]: Done uploading piece. err: <nil> Jul 26 22:17:01.400336069 (25m17.181269454s) [noise=true , address=104.36.82.56:4234 , piece_id=VNGTB2FLRW44V5ZE2TITOPNJYCRTQTGKPQVMUAKTTEZKCAWNZG2A, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400434778 (25m17.181368155s) [noise=true , address=104.36.82.17:4355 , piece_id=YATOCGA6ZHIJ4DMBBJIUWGYIJP2UZSZ76RXYGAZQT2UAEBXITVIA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12mRuEpX7CMxWXVdHrTi3zF3v9t3LcqE6mJKzocajEvYHM52a7T, address:104.36.82.17:4355): protocol: rpc: dial tcp 104.36.82.17:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400496417 (25m17.181429793s) [noise=true , address=104.36.82.11:4138 , piece_id=6HG5FAUP4TNT3DF2E2MOAEDD22LKWM7XQMJ6JUJOFAPEPAQMQKMA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:55266->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400802646 (25m17.181736034s) [noise=true , address=104.36.82.10:4080 , piece_id=4XZAMSGU4QDZSSQOFC5D6EDRLXUWTEPPMUDOUB5INXOIY5CO6G5Q, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3, address:104.36.82.10:4080): protocol: rpc: dial tcp 104.36.82.10:4080: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.400941716 (25m17.181875104s) [noise=true , address=104.36.82.40:4974 , piece_id=XOWD7SGRJGLSUGSX45XC6JFT7IAS3233C4TCYREROUNBRPWIKMJQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:127knbpTBAt5v44Kbz5tYDYfZhmXPa8tBaV43emRXkmKh2E5XMp, address:104.36.82.40:4974): protocol: rpc: dial tcp 104.36.82.40:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401182656 (25m17.182116036s) [noise=true , address=104.36.82.52:4115 , piece_id=TQSWEH44YYCIQA2TXJ7VECTWQPW3NKZ4ELII5RFVXATKUEQJLTRQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401243751 (25m17.182177173s) [noise=true , address=104.36.82.18:4367 , piece_id=UWKE6ZL7I7GDZMCIKRDBOHQRYDFGLRLWHVZTLYWQACTWUAJWKPXA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:104.36.82.18:4367): protocol: rpc: dial tcp 104.36.82.18:4367: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401294101 (25m17.182227463s) [noise=true , address=104.36.82.55:4203 , piece_id=QSGEUM7G42AUEG5JQ6CBAZQCNYNKEOL6TOX7CB5YF2ID4H6I6ICA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU, address:104.36.82.55:4203): protocol: write tcp 159.223.143.39:35722->104.36.82.55:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401346802 (25m17.182280187s) [noise=true , address=104.36.82.53:4135 , piece_id=AMR4IJL6GIO2LWYKZO5NJ4IS3R45DYGLTDSNCKRI3JLP3YZV2ECA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: rpc: dial tcp 104.36.82.53:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401361155 (25m17.182294527s) [noise=true , address=104.36.82.13:4219 , piece_id=SUUXCBWMZ3ZIVQCNBLZU33XZSTKYBDJPNYF7ATZ2ABTFBYPLZEUA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:12fUzLg93yKvyKkNvuvcKZHyvJ2HGE6tUCg1Wm7ZZjKKeDZDWm9, address:104.36.82.13:4219): protocol: rpc: dial tcp 104.36.82.13:4219: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401395307 (25m17.18232867s) [noise=true , address=104.36.82.46:4514 , piece_id=LGVYOFWWLES44276UBMMALIL36Z3FJ3LOPTX5U2OGCHOG4E3B6DQ, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: read tcp 159.223.143.39:52234->104.36.82.46:4514: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401434086 (25m17.18236747s) [noise=true , address=104.36.82.47:4533 , piece_id=N46TAALRQWAKFWWRDTNXL4JHD7LGN7TTFM2H5DXHWPKJUDFCVVBA, seg_pos={0 0}, upload=378]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: read tcp 159.223.143.39:35992->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:01.401502173 (25m17.182435673s) [seg_pos={0 0}, upload=378]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:17:01.430081766 (25m17.211015139s) [upload=378]: Done uploading segments... Jul 26 22:17:01.430115832 (25m17.211049218s) [upload=378]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.033580672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................X..X........XXX.XXX.X.XX..X..X...] duration: 791.779054ms FAIL (2025-07-26 22:16:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 791.907737ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 502.811049ms FAIL (2025-07-26 22:16:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 502.837367ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................XX..XXX..X..XX..X..XX.X....X....] duration: 582.093851ms FAIL (2025-07-26 22:16:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== tocol: rpc: dial tcp 104.36.82.15:4257: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864074530 (20m15.64500791s) [noise=true , address=104.36.82.4:4802 , piece_id=VA26G7K6XQFH4PHZCJ2XWTGOD4GD5JR6DPVR35KCMLMWP4TGBXMA, seg_pos={0 0}, upload=320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:35088->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:11:59.864116163 (20m15.645049543s) [seg_pos={0 0}, upload=320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:59.864361342 (20m15.645294726s) [upload=320]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:59.864371913 (20m15.645305299s) [upload=320]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:59.917691593 (20m15.698624962s) [upload=320]: Done uploading segments... Jul 26 22:11:59.917714191 (20m15.698647562s) [upload=320]: Upload finished. err: <nil> Jul 26 22:12:53.913781892 (21m9.69471528s) [upload=334]: Starting upload Jul 26 22:12:53.913910566 (21m9.694843946s) [upload=334]: Done starting upload Jul 26 22:12:53.914068111 (21m9.695001479s) [upload=334]: Uploading segments... Jul 26 22:12:53.914418778 (21m9.695352164s) [upload=334]: Got next segment. Inline: false Jul 26 22:12:53.914489104 (21m9.695422468s) [upload=334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:53.914499080 (21m9.695432464s) [upload=334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:53.964779415 (21m9.745712792s) [seg_pos={0 0}, upload=334]: Begin upload segment... Jul 26 22:12:53.964834832 (21m9.745768199s) [seg_pos={0 0}, upload=334]: Done begin upload segment. Jul 26 22:12:53.964861055 (21m9.745794419s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.964889604 (21m9.745822981s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965340371 (21m9.746273751s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.965731230 (21m9.746664613s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966122432 (21m9.747055812s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966494352 (21m9.747427739s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.966826509 (21m9.747759885s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967213838 (21m9.748147227s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.967583090 (21m9.748516472s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969295495 (21m9.75022888s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.969752444 (21m9.750685812s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970073562 (21m9.75100693s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.970432919 (21m9.751366305s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.971368537 (21m9.752301923s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.972505536 (21m9.753438924s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:53.973901396 (21m9.754834793s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Uploading piece... Jul 26 22:12:54.096343157 (21m9.877276542s) [upload=334]: Next returned nil segment Jul 26 22:12:54.096365409 (21m9.877298804s) [upload=334]: Waiting for error group managing segments... Jul 26 22:12:54.102545693 (21m9.883479076s) [noise=true , address=45.91.116.8:4047 , piece_id=T6TR77JMONCJKMI6GFYRV5TQ5TKEQK4MFNKY7X4DAOR27R3EP5YA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103199713 (21m9.884133094s) [noise=true , address=45.91.116.14:4191 , piece_id=4NNBN3L7BVHA5RXVW4UIOR6VNYEIJIR6NQJEXPSUICDN5PGVBUPA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103330731 (21m9.884264101s) [noise=true , address=45.91.116.5:4299 , piece_id=XRYRAUFOWUKCRKI6MAOSLBEWQCN3XZ723EP5EHW3MI5LQDK74WYQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.103766692 (21m9.884700057s) [noise=true , address=45.91.116.10:4081 , piece_id=SN2QLCGB7LI4QSA3HZTZ24YN3GKAX73NBSHUQGMGDSQ4S4ALCRNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104301838 (21m9.885235206s) [noise=true , address=45.91.116.12:4227 , piece_id=4YFEHRIMU53ERK6Q6N3JCJAZCPPI763ILSLXQO4ENQ3XCR53XF2A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104452646 (21m9.885386024s) [noise=true , address=45.91.116.14:4189 , piece_id=KYRPA44HXUVQCDM6UC2Y4OH6UD5FPGP73CPWM23OECV2WHLIIV5A, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.104983988 (21m9.885917367s) [noise=true , address=45.91.116.4:4940 , piece_id=PNZIFY2K2JTF3VTT6U72JDFHJQJXR6AXADB5M6TUI76GF7VX6CNQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105066863 (21m9.886000233s) [noise=true , address=45.91.116.13:4155 , piece_id=NO4CW3D2BGKTPZOYQ67QCDJJN4PHJ36PJUTZ7PYRQXDOWMRDDQMA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105555466 (21m9.886488829s) [noise=true , address=77.90.31.16:4734 , piece_id=HGZKZJDEWBNLMHNPK7QWASGK64GK2SCRT2RKGHQRTWV6HNXSGPHQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.105740907 (21m9.886674273s) [noise=true , address=45.91.116.11:4119 , piece_id=HM4JHZT6KIVY5MEZLDSLKUU2C4KYXRVOQWGQ6IS7A5U6V2NBGUXQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108750596 (21m9.889683966s) [noise=true , address=77.90.31.11:4553 , piece_id=RDYZ6MLUNUMTMVBKLJTCRHV27PUFSRQLRSASD4KY6DXPWAY46RLA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:12:54.108767895 (21m9.889701273s) [noise=true , address=77.90.31.14:4661 , piece_id=SUYSFOOETNJRXMR5CVBQJDLYXJK6GH53LBUCVSVT3YOPRYUK3FOA, seg_pos={0 0}, upload=334]: Done uploading piece. err: <nil> Jul 26 22:13:23.875895875 (21m39.656829258s) [noise=true , address=104.36.82.3:4767 , piece_id=VAOTLVRCFBYQLSVSWKD2HJO64QN4GE3FQ32DAADMBJVA7P7YXBHA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875916089 (21m39.656849455s) [noise=true , address=104.36.82.14:4222 , piece_id=PRZV7H3ST54WQWXHAGQMPRJE5PTBHXFULDDPMFSCSYJYCZ3W3CWA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.875996243 (21m39.656929626s) [noise=true , address=104.36.82.9:4005 , piece_id=5BZ4WYO5VJPP6VBFN356ZN5ELRVXFIKYZJOXMWIALC2PZFZHR5GQ, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:33108->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876011671 (21m39.656945039s) [noise=true , address=104.36.82.5:4840 , piece_id=4HO2K6OBVSNMNBWI5QIM64EVS7IZGPUOYCZMSONPLNRRPW4DNKBA, seg_pos={0 0}, upload=334]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:47806->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:13:23.876035419 (21m39.656968803s) [seg_pos={0 0}, upload=334]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:13:23.902382513 (21m39.683315882s) [upload=334]: Done uploading segments... Jul 26 22:13:23.902433922 (21m39.683367306s) [upload=334]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 582.119293ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 15.82698333s FAIL (2025-07-26 22:16:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.827075894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 538.89825ms FAIL (2025-07-26 22:16:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.922738ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)