Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 165025 reports (Page 3101 of 3301)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 17.138868409s FAIL (2025-07-09 02:22:28 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 17.138888954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..........................X.....X...X..............] duration: 530.567965ms FAIL (2025-07-09 02:22:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 530.59782ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 19.490256099s FAIL (2025-07-09 02:22:11 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 19.490308171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...............X....X....X.........X....X.X..XX.XX..X.X.X.] duration: 9.155185722s FAIL (2025-07-09 02:22:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.155211708s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.X] duration: 1m0.001294324s FAIL (2025-07-09 02:21:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001318148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X..........................X.....X...X.............] duration: 8.983274723s FAIL (2025-07-09 02:21:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.983297595s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...............X....X....X.........X....X.X..XX.XX..X.X.X] duration: 30.000545023s FAIL (2025-07-09 02:21:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000565893s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 32.61287081s FAIL (2025-07-09 02:21:17 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 32.612895213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X..........................X.....X...X............] duration: 20.653233006s FAIL (2025-07-09 02:21:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 20.653265249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 33.372527746s FAIL (2025-07-09 02:20:44 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 33.372563505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X..........................X.....X...X...........] duration: 1.057887369s FAIL (2025-07-09 02:20:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.057924288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.] duration: 24.944654815s FAIL (2025-07-09 02:20:19 UTC) unexpected EOF Duration: 24.944680062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X...............X....X....X.........X....X.X..XX.XX..X.X.] duration: 20.345482666s FAIL (2025-07-09 02:20:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.345569254s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [X.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX] duration: 23.478888445s FAIL (2025-07-09 02:19:51 UTC) unexpected EOF Duration: 23.478905286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X...............X....X....X.........X....X.X..XX.XX..X.X] duration: 30.001271503s FAIL (2025-07-09 02:19:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001298804s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X..........................X.....X...X..........] duration: 828.635186ms FAIL (2025-07-09 02:19:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 828.683198ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXX] duration: 1m0.001302148s FAIL (2025-07-09 02:19:28 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00132223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 22.305909192s FAIL (2025-07-09 02:19:19 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 22.305943189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X...............X....X....X.........X....X.X..XX.XX..X.] duration: 24.114431974s FAIL (2025-07-09 02:18:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.114445865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 27.707977305s FAIL (2025-07-09 02:18:25 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 27.708010747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X..........................X.....X...X.........] duration: 434.600408ms FAIL (2025-07-09 02:18:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.625377ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [.XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXX] duration: 1m0.000902635s FAIL (2025-07-09 02:18:19 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000916956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X...............X....X....X.........X....X.X..XX.XX..X] duration: 30.001281742s FAIL (2025-07-09 02:18:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001301118s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 23.83951535s FAIL (2025-07-09 02:17:44 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 23.839543559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 39.193457991s FAIL (2025-07-09 02:17:20 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 39.193509278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXX] duration: 1m0.001913577s FAIL (2025-07-09 02:17:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001951187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X..........................X.....X...X........] duration: 865.923683ms FAIL (2025-07-09 02:17:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 865.952151ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X......X...............X....X....X.........X....X.X..XX.XX..] duration: 9.752345296s FAIL (2025-07-09 02:17:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.752374421s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X......X...............X....X....X.........X....X.X..XX.XX.] duration: 12.320666438s FAIL (2025-07-09 02:17:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.32071324s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X......X...............X....X....X.........X....X.X..XX.XX] duration: 30.00147128s FAIL (2025-07-09 02:16:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001503921s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X..........................X.....X...X.......] duration: 430.682882ms FAIL (2025-07-09 02:16:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.721026ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X......X...............X....X....X.........X....X.X..XX.X] duration: 30.0009714s FAIL (2025-07-09 02:16:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001000088s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [X..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XX] duration: 1m0.000552779s FAIL (2025-07-09 02:16:12 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000571559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.933122822s FAIL (2025-07-09 02:16:10 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 15.933149582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000565808s FAIL (2025-07-09 02:15:54 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 1m0.000593317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X..........................X.....X...X......] duration: 533.869094ms FAIL (2025-07-09 02:15:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 533.902192ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.X] duration: 1m0.001505506s FAIL (2025-07-09 02:15:12 UTC) context deadline exceeded Duration: 1m0.001524493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X......X...............X....X....X.........X....X.X..XX.] duration: 9.507371717s FAIL (2025-07-09 02:15:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.507944871s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X......X...............X....X....X.........X....X.X..XX] duration: 30.000846794s FAIL (2025-07-09 02:14:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000869236s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X..........................X.....X...X.....] duration: 555.740788ms FAIL (2025-07-09 02:14:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 555.773722ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [.XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.] duration: 51.663188902s FAIL (2025-07-09 02:14:12 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.6632095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X......X...............X....X....X.........X....X.X..X] duration: 30.001768794s FAIL (2025-07-09 02:14:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001816391s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X..........................X.....X...X....] duration: 14.702210217s FAIL (2025-07-09 02:13:52 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.70223944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX] duration: 1m0.000505762s FAIL (2025-07-09 02:13:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000821078s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X......X......X...............X....X....X.........X....X.X..] duration: 716.709478ms FAIL (2025-07-09 02:13:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 716.73823ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X..........................X.....X...X...] duration: 23.099578362s FAIL (2025-07-09 02:13:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.099607924s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X......X......X...............X....X....X.........X....X.X.] duration: 981.38836ms FAIL (2025-07-09 02:12:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 981.416115ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.X......X......X...............X....X....X.........X....X.X] duration: 30.001163026s FAIL (2025-07-09 02:12:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00119037s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.X] duration: 1m0.000776577s FAIL (2025-07-09 02:12:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000800156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X..........................X.....X...X..] duration: 424.441397ms FAIL (2025-07-09 02:11:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 424.472445ms core uplink us-select-1

1
...
3099
3100
3101
3102
3103
...
3301