Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [.X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.] duration: 44.108214198s FAIL (2025-07-09 02:11:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.10822842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.X......X......X...............X....X....X.........X....X.] duration: 8.661032203s FAIL (2025-07-09 02:11:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.661051163s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X..........................X.....X...X.] duration: 28.02939259s FAIL (2025-07-09 02:11:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 28.029421201s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.X.X......X......X...............X....X....X.........X....X] duration: 30.000908029s FAIL (2025-07-09 02:11:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000946871s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX] duration: 1m0.000473483s FAIL (2025-07-09 02:10:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000483979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X..........................X.....X...X] duration: 30.044308306s FAIL (2025-07-09 02:10:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.044362442s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.X.X......X......X...............X....X....X.........X....] duration: 7.361796195s FAIL (2025-07-09 02:10:07 UTC) orbiter warning: context deadline exceeded Duration: 7.361817192s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [X..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XX] duration: 1m0.000783247s FAIL (2025-07-09 02:09:33 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000817903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.X.X......X......X...............X....X....X.........X...] duration: 1.404627484s FAIL (2025-07-09 02:09:05 UTC) orbiter warning: context deadline exceeded Duration: 1.404736808s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..........................X.....X...] duration: 390.25271ms FAIL (2025-07-09 02:08:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Jul 9 02:05:42.380732414 (376h54m9.307027091s) [seg_pos={0 0}, upload=301848]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 9 02:05:42.380893823 (376h54m9.307188524s) [upload=301848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:05:42.380913392 (376h54m9.307208095s) [upload=301848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:05:42.442439839 (376h54m9.368734525s) [upload=301848]: Done uploading segments... Jul 9 02:05:42.442457702 (376h54m9.368752379s) [upload=301848]: Upload finished. err: <nil> Jul 9 02:06:26.900176697 (376h54m53.826471384s) [upload=301859]: Starting upload Jul 9 02:06:26.900265388 (376h54m53.826560077s) [upload=301859]: Done starting upload Jul 9 02:06:26.900652891 (376h54m53.826947566s) [upload=301859]: Uploading segments... Jul 9 02:06:26.900704500 (376h54m53.826999202s) [upload=301859]: Got next segment. Inline: false Jul 9 02:06:26.900741590 (376h54m53.827036278s) [upload=301859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:06:26.900749288 (376h54m53.827043976s) [upload=301859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:06:26.927751705 (376h54m53.854046403s) [seg_pos={0 0}, upload=301859]: Begin upload segment... Jul 9 02:06:26.927838738 (376h54m53.854133428s) [seg_pos={0 0}, upload=301859]: Done begin upload segment. Jul 9 02:06:26.927863464 (376h54m53.854158166s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.927886021 (376h54m53.854180718s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928433445 (376h54m53.854728142s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928586672 (376h54m53.854881371s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928859328 (376h54m53.85515401s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929261055 (376h54m53.855555753s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929930444 (376h54m53.856225145s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.930491886 (376h54m53.856786583s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931181441 (376h54m53.857476139s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931641078 (376h54m53.857935775s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932051000 (376h54m53.858345694s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932205873 (376h54m53.858500571s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932921617 (376h54m53.859216396s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933283250 (376h54m53.859577944s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933741894 (376h54m53.860036589s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.934191911 (376h54m53.860486608s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:27.027837362 (376h54m53.954132039s) [upload=301859]: Next returned nil segment Jul 9 02:06:27.027852248 (376h54m53.954146934s) [upload=301859]: Waiting for error group managing segments... Jul 9 02:06:27.031627567 (376h54m53.957922255s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.031656202 (376h54m53.957950875s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032312286 (376h54m53.958606964s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032359743 (376h54m53.958654431s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032906233 (376h54m53.959200907s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032958030 (376h54m53.95925272s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033424093 (376h54m53.959718768s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033555193 (376h54m53.959849866s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.035104256 (376h54m53.961398947s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.037389678 (376h54m53.963684359s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.351061544 (376h54m54.277356236s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:48.505779764 (376h55m15.432074452s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:56.865619467 (376h55m23.791914165s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865737493 (376h55m23.792032179s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection Jul 9 02:06:56.865776568 (376h55m23.792071244s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865828470 (376h55m23.79212315s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection Jul 9 02:06:56.865849177 (376h55m23.792143863s) [seg_pos={0 0}, upload=301859]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:06:56.891096255 (376h55m23.81739094s) [upload=301859]: Done uploading segments... Jul 9 02:06:56.891129386 (376h55m23.81742408s) [upload=301859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.284498ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.X] duration: 1m0.001655353s FAIL (2025-07-09 02:08:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001676407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.X.X......X......X...............X....X....X.........X..] duration: 7.937187988s FAIL (2025-07-09 02:08:21 UTC) orbiter warning: context deadline exceeded Duration: 7.937227072s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X..........................X.....X..] duration: 10.548208014s FAIL (2025-07-09 02:08:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Jul 9 02:05:42.380732414 (376h54m9.307027091s) [seg_pos={0 0}, upload=301848]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 9 02:05:42.380893823 (376h54m9.307188524s) [upload=301848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:05:42.380913392 (376h54m9.307208095s) [upload=301848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:05:42.442439839 (376h54m9.368734525s) [upload=301848]: Done uploading segments... Jul 9 02:05:42.442457702 (376h54m9.368752379s) [upload=301848]: Upload finished. err: <nil> Jul 9 02:06:26.900176697 (376h54m53.826471384s) [upload=301859]: Starting upload Jul 9 02:06:26.900265388 (376h54m53.826560077s) [upload=301859]: Done starting upload Jul 9 02:06:26.900652891 (376h54m53.826947566s) [upload=301859]: Uploading segments... Jul 9 02:06:26.900704500 (376h54m53.826999202s) [upload=301859]: Got next segment. Inline: false Jul 9 02:06:26.900741590 (376h54m53.827036278s) [upload=301859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:06:26.900749288 (376h54m53.827043976s) [upload=301859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:06:26.927751705 (376h54m53.854046403s) [seg_pos={0 0}, upload=301859]: Begin upload segment... Jul 9 02:06:26.927838738 (376h54m53.854133428s) [seg_pos={0 0}, upload=301859]: Done begin upload segment. Jul 9 02:06:26.927863464 (376h54m53.854158166s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.927886021 (376h54m53.854180718s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928433445 (376h54m53.854728142s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928586672 (376h54m53.854881371s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928859328 (376h54m53.85515401s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929261055 (376h54m53.855555753s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929930444 (376h54m53.856225145s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.930491886 (376h54m53.856786583s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931181441 (376h54m53.857476139s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931641078 (376h54m53.857935775s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932051000 (376h54m53.858345694s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932205873 (376h54m53.858500571s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932921617 (376h54m53.859216396s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933283250 (376h54m53.859577944s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933741894 (376h54m53.860036589s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.934191911 (376h54m53.860486608s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:27.027837362 (376h54m53.954132039s) [upload=301859]: Next returned nil segment Jul 9 02:06:27.027852248 (376h54m53.954146934s) [upload=301859]: Waiting for error group managing segments... Jul 9 02:06:27.031627567 (376h54m53.957922255s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.031656202 (376h54m53.957950875s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032312286 (376h54m53.958606964s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032359743 (376h54m53.958654431s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032906233 (376h54m53.959200907s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032958030 (376h54m53.95925272s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033424093 (376h54m53.959718768s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033555193 (376h54m53.959849866s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.035104256 (376h54m53.961398947s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.037389678 (376h54m53.963684359s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.351061544 (376h54m54.277356236s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:48.505779764 (376h55m15.432074452s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:56.865619467 (376h55m23.791914165s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865737493 (376h55m23.792032179s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection Jul 9 02:06:56.865776568 (376h55m23.792071244s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865828470 (376h55m23.79212315s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection Jul 9 02:06:56.865849177 (376h55m23.792143863s) [seg_pos={0 0}, upload=301859]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:06:56.891096255 (376h55m23.81739094s) [upload=301859]: Done uploading segments... Jul 9 02:06:56.891129386 (376h55m23.81742408s) [upload=301859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.548234371s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.X.X......X......X...............X....X....X.........X.] duration: 26.997084492s FAIL (2025-07-09 02:07:48 UTC) orbiter warning: context deadline exceeded Duration: 26.997113727s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.] duration: 41.459585747s FAIL (2025-07-09 02:07:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.459607278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....X.X.X......X......X...............X....X....X.........X] duration: 30.000988693s FAIL (2025-07-09 02:07:16 UTC) orbiter warning: context deadline exceeded Duration: 30.00101478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..........................X.....X.] duration: 470.878313ms FAIL (2025-07-09 02:07:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Jul 9 02:05:42.380732414 (376h54m9.307027091s) [seg_pos={0 0}, upload=301848]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 9 02:05:42.380893823 (376h54m9.307188524s) [upload=301848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:05:42.380913392 (376h54m9.307208095s) [upload=301848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:05:42.442439839 (376h54m9.368734525s) [upload=301848]: Done uploading segments... Jul 9 02:05:42.442457702 (376h54m9.368752379s) [upload=301848]: Upload finished. err: <nil> Jul 9 02:06:26.900176697 (376h54m53.826471384s) [upload=301859]: Starting upload Jul 9 02:06:26.900265388 (376h54m53.826560077s) [upload=301859]: Done starting upload Jul 9 02:06:26.900652891 (376h54m53.826947566s) [upload=301859]: Uploading segments... Jul 9 02:06:26.900704500 (376h54m53.826999202s) [upload=301859]: Got next segment. Inline: false Jul 9 02:06:26.900741590 (376h54m53.827036278s) [upload=301859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:06:26.900749288 (376h54m53.827043976s) [upload=301859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:06:26.927751705 (376h54m53.854046403s) [seg_pos={0 0}, upload=301859]: Begin upload segment... Jul 9 02:06:26.927838738 (376h54m53.854133428s) [seg_pos={0 0}, upload=301859]: Done begin upload segment. Jul 9 02:06:26.927863464 (376h54m53.854158166s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.927886021 (376h54m53.854180718s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928433445 (376h54m53.854728142s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928586672 (376h54m53.854881371s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928859328 (376h54m53.85515401s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929261055 (376h54m53.855555753s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929930444 (376h54m53.856225145s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.930491886 (376h54m53.856786583s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931181441 (376h54m53.857476139s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931641078 (376h54m53.857935775s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932051000 (376h54m53.858345694s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932205873 (376h54m53.858500571s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932921617 (376h54m53.859216396s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933283250 (376h54m53.859577944s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933741894 (376h54m53.860036589s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.934191911 (376h54m53.860486608s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:27.027837362 (376h54m53.954132039s) [upload=301859]: Next returned nil segment Jul 9 02:06:27.027852248 (376h54m53.954146934s) [upload=301859]: Waiting for error group managing segments... Jul 9 02:06:27.031627567 (376h54m53.957922255s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.031656202 (376h54m53.957950875s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032312286 (376h54m53.958606964s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032359743 (376h54m53.958654431s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032906233 (376h54m53.959200907s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032958030 (376h54m53.95925272s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033424093 (376h54m53.959718768s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033555193 (376h54m53.959849866s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.035104256 (376h54m53.961398947s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.037389678 (376h54m53.963684359s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.351061544 (376h54m54.277356236s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:48.505779764 (376h55m15.432074452s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:56.865619467 (376h55m23.791914165s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865737493 (376h55m23.792032179s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection Jul 9 02:06:56.865776568 (376h55m23.792071244s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865828470 (376h55m23.79212315s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection Jul 9 02:06:56.865849177 (376h55m23.792143863s) [seg_pos={0 0}, upload=301859]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:06:56.891096255 (376h55m23.81739094s) [upload=301859]: Done uploading segments... Jul 9 02:06:56.891129386 (376h55m23.81742408s) [upload=301859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.905419ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..........................X.....X] duration: 30.027118034s FAIL (2025-07-09 02:06:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Jul 9 02:05:42.380732414 (376h54m9.307027091s) [seg_pos={0 0}, upload=301848]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 9 02:05:42.380893823 (376h54m9.307188524s) [upload=301848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:05:42.380913392 (376h54m9.307208095s) [upload=301848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:05:42.442439839 (376h54m9.368734525s) [upload=301848]: Done uploading segments... Jul 9 02:05:42.442457702 (376h54m9.368752379s) [upload=301848]: Upload finished. err: <nil> Jul 9 02:06:26.900176697 (376h54m53.826471384s) [upload=301859]: Starting upload Jul 9 02:06:26.900265388 (376h54m53.826560077s) [upload=301859]: Done starting upload Jul 9 02:06:26.900652891 (376h54m53.826947566s) [upload=301859]: Uploading segments... Jul 9 02:06:26.900704500 (376h54m53.826999202s) [upload=301859]: Got next segment. Inline: false Jul 9 02:06:26.900741590 (376h54m53.827036278s) [upload=301859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:06:26.900749288 (376h54m53.827043976s) [upload=301859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:06:26.927751705 (376h54m53.854046403s) [seg_pos={0 0}, upload=301859]: Begin upload segment... Jul 9 02:06:26.927838738 (376h54m53.854133428s) [seg_pos={0 0}, upload=301859]: Done begin upload segment. Jul 9 02:06:26.927863464 (376h54m53.854158166s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.927886021 (376h54m53.854180718s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928433445 (376h54m53.854728142s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928586672 (376h54m53.854881371s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.928859328 (376h54m53.85515401s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929261055 (376h54m53.855555753s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.929930444 (376h54m53.856225145s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.930491886 (376h54m53.856786583s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931181441 (376h54m53.857476139s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.931641078 (376h54m53.857935775s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932051000 (376h54m53.858345694s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932205873 (376h54m53.858500571s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.932921617 (376h54m53.859216396s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933283250 (376h54m53.859577944s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.933741894 (376h54m53.860036589s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:26.934191911 (376h54m53.860486608s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Uploading piece... Jul 9 02:06:27.027837362 (376h54m53.954132039s) [upload=301859]: Next returned nil segment Jul 9 02:06:27.027852248 (376h54m53.954146934s) [upload=301859]: Waiting for error group managing segments... Jul 9 02:06:27.031627567 (376h54m53.957922255s) [noise=true , address=45.91.116.7:4011 , piece_id=XYYJP33EBD2SSRV7MYJJPP3ZQQCZ2SGZGXN254H34ATCSTZHBXXA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.031656202 (376h54m53.957950875s) [noise=true , address=45.91.116.12:4227 , piece_id=QMF4HGMBPRCHACM3ILWT77GAWJFKFNWQRTJTYHUMPBIM4QOBGC5Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032312286 (376h54m53.958606964s) [noise=true , address=45.91.116.10:4082 , piece_id=7GDNLBYHCOYQICYTN5XH3J5HH55TBRBVY6VJ7O7QLIRHTZEDGXYA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032359743 (376h54m53.958654431s) [noise=true , address=45.91.116.9:4261 , piece_id=NYT2JRGZQGYCGZ6IY5BVILVLMQ27MW57GWP7EDJDXVHNFNFWZNQA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032906233 (376h54m53.959200907s) [noise=true , address=45.91.116.4:4939 , piece_id=NISNFKWHMGLP4RXQYFP3FIATXAOXTBVA4NCCYRIQCSOVX3JXFJEQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.032958030 (376h54m53.95925272s) [noise=true , address=45.91.116.8:4045 , piece_id=5JB5ZEBHDB6C3DQUK5I72UFSIH7A5J5BCS4MFV5TKCTT5RLOILFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033424093 (376h54m53.959718768s) [noise=true , address=45.91.116.13:4154 , piece_id=64GHNRFN6BADGNAPXEVA7DSKD26UZJG45OL75Z64KLQD4W7W6OIQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.033555193 (376h54m53.959849866s) [noise=true , address=45.91.116.14:4189 , piece_id=MH5RRU7UKK2ALTE4VWDPGGL35XC4KBWSBKEXDEW2XXI4UZYWMSMA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.035104256 (376h54m53.961398947s) [noise=true , address=77.90.31.10:4518 , piece_id=O5FSMM4OV2S3MBL6IEYBG6NNJ3PF5X7KK6PRFRPTJ5UOHE4TNJGA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.037389678 (376h54m53.963684359s) [noise=true , address=104.36.82.9:4007 , piece_id=CDQZ34F6IYSDMOJP4ROY5VXOPT5YWNEX22W2FIZPAIJPIMADR6TQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:27.351061544 (376h54m54.277356236s) [noise=true , address=45.91.116.14:4190 , piece_id=IQNTCZ5MXZ63GRFA63CMFFWIYUEXT37C5RSONEVKZD3LMEI3DZ3Q, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:48.505779764 (376h55m15.432074452s) [noise=true , address=104.36.82.15:4259 , piece_id=MEW37RHLSN4PQZLRW2364Y6M2UXKA5QZF7SH4TNRHSN2AWYLEL6A, seg_pos={0 0}, upload=301859]: Done uploading piece. err: <nil> Jul 9 02:06:56.865619467 (376h55m23.791914165s) [noise=true , address=104.36.82.7:4911 , piece_id=IZEJROIQ7TEO3GSSW4VQP2F4K7X33I4MROCF26VJYVXKG3RW5NFQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865737493 (376h55m23.792032179s) [noise=true , address=104.36.82.6:4874 , piece_id=FTPQG2QAEZXP3V6USN3IYTTW6RDVY2MRLNQ5JO3ILNENPMNYTILA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45756->104.36.82.6:4874: use of closed network connection Jul 9 02:06:56.865776568 (376h55m23.792071244s) [noise=true , address=104.36.82.12:4150 , piece_id=P2JNVDC2DZLGSFFZ2HJRV6T6TFYBDP2WWKVUG3YTS6F7LH6NVOVA, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:06:56.865828470 (376h55m23.79212315s) [noise=true , address=104.36.82.8:4969 , piece_id=QNC4SQSLIJH4ZM2WONGV2WOHQJ2EEJQKHCVTM2NABRL5EUFVXNUQ, seg_pos={0 0}, upload=301859]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60844->104.36.82.8:4969: use of closed network connection Jul 9 02:06:56.865849177 (376h55m23.792143863s) [seg_pos={0 0}, upload=301859]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:06:56.891096255 (376h55m23.81739094s) [upload=301859]: Done uploading segments... Jul 9 02:06:56.891129386 (376h55m23.81742408s) [upload=301859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02714356s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [..XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX] duration: 1m0.001110875s FAIL (2025-07-09 02:06:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001133519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X....X.X.X......X......X...............X....X....X.........] duration: 7.49271084s FAIL (2025-07-09 02:06:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.493356867s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..........................X.....] duration: 463.48091ms FAIL (2025-07-09 02:05:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== os={0 0}, upload=301798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:56758->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 02:01:52.739421960 (376h50m19.665716638s) [seg_pos={0 0}, upload=301798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:01:52.739490843 (376h50m19.665785529s) [upload=301798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:01:52.739495541 (376h50m19.665790227s) [upload=301798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:01:52.835070223 (376h50m19.761364899s) [upload=301798]: Done uploading segments... Jul 9 02:01:52.835100626 (376h50m19.761395302s) [upload=301798]: Upload finished. err: <nil> Jul 9 02:02:41.019556268 (376h51m7.945850961s) [upload=301809]: Starting upload Jul 9 02:02:41.019637243 (376h51m7.945931929s) [upload=301809]: Done starting upload Jul 9 02:02:41.019785898 (376h51m7.946080574s) [upload=301809]: Uploading segments... Jul 9 02:02:41.019931314 (376h51m7.946225987s) [upload=301809]: Got next segment. Inline: false Jul 9 02:02:41.020015521 (376h51m7.946310209s) [upload=301809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:02:41.020021323 (376h51m7.946315999s) [upload=301809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:02:41.047332594 (376h51m7.973627288s) [seg_pos={0 0}, upload=301809]: Begin upload segment... Jul 9 02:02:41.047410558 (376h51m7.973705232s) [seg_pos={0 0}, upload=301809]: Done begin upload segment. Jul 9 02:02:41.047474424 (376h51m7.973769121s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048070277 (376h51m7.974364973s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048621968 (376h51m7.974916664s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049099340 (376h51m7.975394033s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049624028 (376h51m7.97593493s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050041504 (376h51m7.97633619s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050418785 (376h51m7.976713485s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050860042 (376h51m7.977154734s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051563500 (376h51m7.977858192s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051963243 (376h51m7.978257933s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052302595 (376h51m7.978597349s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052672504 (376h51m7.978967197s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053100417 (376h51m7.979395114s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053445269 (376h51m7.979739955s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053883256 (376h51m7.980177948s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.054279476 (376h51m7.980574162s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.130875320 (376h51m8.057170014s) [upload=301809]: Next returned nil segment Jul 9 02:02:41.130895759 (376h51m8.057190457s) [upload=301809]: Waiting for error group managing segments... Jul 9 02:02:41.136204286 (376h51m8.062498966s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.136645242 (376h51m8.062939934s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137355020 (376h51m8.063649707s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137902951 (376h51m8.064197633s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139285699 (376h51m8.065580372s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139410254 (376h51m8.06570494s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141152669 (376h51m8.067447346s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141684412 (376h51m8.067979085s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.142262736 (376h51m8.068557427s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.145949224 (376h51m8.072243912s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:56.644337945 (376h51m23.570632617s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:00.255623941 (376h51m27.18191863s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:10.988827693 (376h51m37.915122369s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988897302 (376h51m37.915191981s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988913335 (376h51m37.915208027s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.989022369 (376h51m37.915317066s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection Jul 9 02:03:10.989097572 (376h51m37.915392265s) [seg_pos={0 0}, upload=301809]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:03:11.020835208 (376h51m37.947129912s) [upload=301809]: Done uploading segments... Jul 9 02:03:11.020880111 (376h51m37.9471748s) [upload=301809]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.509757ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [...XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XX] duration: 1m0.000797654s FAIL (2025-07-09 02:05:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000810927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..........................X....] duration: 772.444002ms FAIL (2025-07-09 02:05:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== os={0 0}, upload=301798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:56758->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 02:01:52.739421960 (376h50m19.665716638s) [seg_pos={0 0}, upload=301798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:01:52.739490843 (376h50m19.665785529s) [upload=301798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:01:52.739495541 (376h50m19.665790227s) [upload=301798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:01:52.835070223 (376h50m19.761364899s) [upload=301798]: Done uploading segments... Jul 9 02:01:52.835100626 (376h50m19.761395302s) [upload=301798]: Upload finished. err: <nil> Jul 9 02:02:41.019556268 (376h51m7.945850961s) [upload=301809]: Starting upload Jul 9 02:02:41.019637243 (376h51m7.945931929s) [upload=301809]: Done starting upload Jul 9 02:02:41.019785898 (376h51m7.946080574s) [upload=301809]: Uploading segments... Jul 9 02:02:41.019931314 (376h51m7.946225987s) [upload=301809]: Got next segment. Inline: false Jul 9 02:02:41.020015521 (376h51m7.946310209s) [upload=301809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:02:41.020021323 (376h51m7.946315999s) [upload=301809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:02:41.047332594 (376h51m7.973627288s) [seg_pos={0 0}, upload=301809]: Begin upload segment... Jul 9 02:02:41.047410558 (376h51m7.973705232s) [seg_pos={0 0}, upload=301809]: Done begin upload segment. Jul 9 02:02:41.047474424 (376h51m7.973769121s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048070277 (376h51m7.974364973s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048621968 (376h51m7.974916664s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049099340 (376h51m7.975394033s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049624028 (376h51m7.97593493s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050041504 (376h51m7.97633619s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050418785 (376h51m7.976713485s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050860042 (376h51m7.977154734s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051563500 (376h51m7.977858192s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051963243 (376h51m7.978257933s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052302595 (376h51m7.978597349s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052672504 (376h51m7.978967197s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053100417 (376h51m7.979395114s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053445269 (376h51m7.979739955s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053883256 (376h51m7.980177948s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.054279476 (376h51m7.980574162s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.130875320 (376h51m8.057170014s) [upload=301809]: Next returned nil segment Jul 9 02:02:41.130895759 (376h51m8.057190457s) [upload=301809]: Waiting for error group managing segments... Jul 9 02:02:41.136204286 (376h51m8.062498966s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.136645242 (376h51m8.062939934s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137355020 (376h51m8.063649707s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137902951 (376h51m8.064197633s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139285699 (376h51m8.065580372s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139410254 (376h51m8.06570494s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141152669 (376h51m8.067447346s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141684412 (376h51m8.067979085s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.142262736 (376h51m8.068557427s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.145949224 (376h51m8.072243912s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:56.644337945 (376h51m23.570632617s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:00.255623941 (376h51m27.18191863s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:10.988827693 (376h51m37.915122369s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988897302 (376h51m37.915191981s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988913335 (376h51m37.915208027s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.989022369 (376h51m37.915317066s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection Jul 9 02:03:10.989097572 (376h51m37.915392265s) [seg_pos={0 0}, upload=301809]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:03:11.020835208 (376h51m37.947129912s) [upload=301809]: Done uploading segments... Jul 9 02:03:11.020880111 (376h51m37.9471748s) [upload=301809]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 772.470739ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..........................X...] duration: 879.373946ms FAIL (2025-07-09 02:05:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== os={0 0}, upload=301798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:56758->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 02:01:52.739421960 (376h50m19.665716638s) [seg_pos={0 0}, upload=301798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:01:52.739490843 (376h50m19.665785529s) [upload=301798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:01:52.739495541 (376h50m19.665790227s) [upload=301798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:01:52.835070223 (376h50m19.761364899s) [upload=301798]: Done uploading segments... Jul 9 02:01:52.835100626 (376h50m19.761395302s) [upload=301798]: Upload finished. err: <nil> Jul 9 02:02:41.019556268 (376h51m7.945850961s) [upload=301809]: Starting upload Jul 9 02:02:41.019637243 (376h51m7.945931929s) [upload=301809]: Done starting upload Jul 9 02:02:41.019785898 (376h51m7.946080574s) [upload=301809]: Uploading segments... Jul 9 02:02:41.019931314 (376h51m7.946225987s) [upload=301809]: Got next segment. Inline: false Jul 9 02:02:41.020015521 (376h51m7.946310209s) [upload=301809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:02:41.020021323 (376h51m7.946315999s) [upload=301809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:02:41.047332594 (376h51m7.973627288s) [seg_pos={0 0}, upload=301809]: Begin upload segment... Jul 9 02:02:41.047410558 (376h51m7.973705232s) [seg_pos={0 0}, upload=301809]: Done begin upload segment. Jul 9 02:02:41.047474424 (376h51m7.973769121s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048070277 (376h51m7.974364973s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048621968 (376h51m7.974916664s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049099340 (376h51m7.975394033s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049624028 (376h51m7.97593493s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050041504 (376h51m7.97633619s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050418785 (376h51m7.976713485s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050860042 (376h51m7.977154734s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051563500 (376h51m7.977858192s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051963243 (376h51m7.978257933s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052302595 (376h51m7.978597349s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052672504 (376h51m7.978967197s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053100417 (376h51m7.979395114s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053445269 (376h51m7.979739955s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053883256 (376h51m7.980177948s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.054279476 (376h51m7.980574162s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.130875320 (376h51m8.057170014s) [upload=301809]: Next returned nil segment Jul 9 02:02:41.130895759 (376h51m8.057190457s) [upload=301809]: Waiting for error group managing segments... Jul 9 02:02:41.136204286 (376h51m8.062498966s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.136645242 (376h51m8.062939934s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137355020 (376h51m8.063649707s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137902951 (376h51m8.064197633s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139285699 (376h51m8.065580372s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139410254 (376h51m8.06570494s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141152669 (376h51m8.067447346s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141684412 (376h51m8.067979085s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.142262736 (376h51m8.068557427s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.145949224 (376h51m8.072243912s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:56.644337945 (376h51m23.570632617s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:00.255623941 (376h51m27.18191863s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:10.988827693 (376h51m37.915122369s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988897302 (376h51m37.915191981s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988913335 (376h51m37.915208027s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.989022369 (376h51m37.915317066s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection Jul 9 02:03:10.989097572 (376h51m37.915392265s) [seg_pos={0 0}, upload=301809]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:03:11.020835208 (376h51m37.947129912s) [upload=301809]: Done uploading segments... Jul 9 02:03:11.020880111 (376h51m37.9471748s) [upload=301809]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 879.426996ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X....X.X.X......X......X...............X....X....X........] duration: 505.41988ms FAIL (2025-07-09 02:04:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 505.446918ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.X] duration: 1m0.001485284s FAIL (2025-07-09 02:04:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001504034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..........................X..] duration: 11.555171206s FAIL (2025-07-09 02:04:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== os={0 0}, upload=301798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:56758->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 02:01:52.739421960 (376h50m19.665716638s) [seg_pos={0 0}, upload=301798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:01:52.739490843 (376h50m19.665785529s) [upload=301798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:01:52.739495541 (376h50m19.665790227s) [upload=301798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:01:52.835070223 (376h50m19.761364899s) [upload=301798]: Done uploading segments... Jul 9 02:01:52.835100626 (376h50m19.761395302s) [upload=301798]: Upload finished. err: <nil> Jul 9 02:02:41.019556268 (376h51m7.945850961s) [upload=301809]: Starting upload Jul 9 02:02:41.019637243 (376h51m7.945931929s) [upload=301809]: Done starting upload Jul 9 02:02:41.019785898 (376h51m7.946080574s) [upload=301809]: Uploading segments... Jul 9 02:02:41.019931314 (376h51m7.946225987s) [upload=301809]: Got next segment. Inline: false Jul 9 02:02:41.020015521 (376h51m7.946310209s) [upload=301809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:02:41.020021323 (376h51m7.946315999s) [upload=301809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:02:41.047332594 (376h51m7.973627288s) [seg_pos={0 0}, upload=301809]: Begin upload segment... Jul 9 02:02:41.047410558 (376h51m7.973705232s) [seg_pos={0 0}, upload=301809]: Done begin upload segment. Jul 9 02:02:41.047474424 (376h51m7.973769121s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048070277 (376h51m7.974364973s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048621968 (376h51m7.974916664s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049099340 (376h51m7.975394033s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049624028 (376h51m7.97593493s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050041504 (376h51m7.97633619s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050418785 (376h51m7.976713485s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050860042 (376h51m7.977154734s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051563500 (376h51m7.977858192s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051963243 (376h51m7.978257933s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052302595 (376h51m7.978597349s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052672504 (376h51m7.978967197s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053100417 (376h51m7.979395114s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053445269 (376h51m7.979739955s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053883256 (376h51m7.980177948s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.054279476 (376h51m7.980574162s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.130875320 (376h51m8.057170014s) [upload=301809]: Next returned nil segment Jul 9 02:02:41.130895759 (376h51m8.057190457s) [upload=301809]: Waiting for error group managing segments... Jul 9 02:02:41.136204286 (376h51m8.062498966s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.136645242 (376h51m8.062939934s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137355020 (376h51m8.063649707s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137902951 (376h51m8.064197633s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139285699 (376h51m8.065580372s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139410254 (376h51m8.06570494s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141152669 (376h51m8.067447346s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141684412 (376h51m8.067979085s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.142262736 (376h51m8.068557427s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.145949224 (376h51m8.072243912s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:56.644337945 (376h51m23.570632617s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:00.255623941 (376h51m27.18191863s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:10.988827693 (376h51m37.915122369s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988897302 (376h51m37.915191981s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988913335 (376h51m37.915208027s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.989022369 (376h51m37.915317066s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection Jul 9 02:03:10.989097572 (376h51m37.915392265s) [seg_pos={0 0}, upload=301809]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:03:11.020835208 (376h51m37.947129912s) [upload=301809]: Done uploading segments... Jul 9 02:03:11.020880111 (376h51m37.9471748s) [upload=301809]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.555196735s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..X....X.X.X......X......X...............X....X....X.......] duration: 14.049067648s FAIL (2025-07-09 02:04:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.049131839s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.] duration: 12.800121967s FAIL (2025-07-09 02:03:13 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.800175247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..........................X.] duration: 618.99971ms FAIL (2025-07-09 02:03:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== os={0 0}, upload=301798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 159.223.143.39:56758->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 9 02:01:52.739421960 (376h50m19.665716638s) [seg_pos={0 0}, upload=301798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:01:52.739490843 (376h50m19.665785529s) [upload=301798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:01:52.739495541 (376h50m19.665790227s) [upload=301798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:01:52.835070223 (376h50m19.761364899s) [upload=301798]: Done uploading segments... Jul 9 02:01:52.835100626 (376h50m19.761395302s) [upload=301798]: Upload finished. err: <nil> Jul 9 02:02:41.019556268 (376h51m7.945850961s) [upload=301809]: Starting upload Jul 9 02:02:41.019637243 (376h51m7.945931929s) [upload=301809]: Done starting upload Jul 9 02:02:41.019785898 (376h51m7.946080574s) [upload=301809]: Uploading segments... Jul 9 02:02:41.019931314 (376h51m7.946225987s) [upload=301809]: Got next segment. Inline: false Jul 9 02:02:41.020015521 (376h51m7.946310209s) [upload=301809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:02:41.020021323 (376h51m7.946315999s) [upload=301809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:02:41.047332594 (376h51m7.973627288s) [seg_pos={0 0}, upload=301809]: Begin upload segment... Jul 9 02:02:41.047410558 (376h51m7.973705232s) [seg_pos={0 0}, upload=301809]: Done begin upload segment. Jul 9 02:02:41.047474424 (376h51m7.973769121s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048070277 (376h51m7.974364973s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.048621968 (376h51m7.974916664s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049099340 (376h51m7.975394033s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.049624028 (376h51m7.97593493s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050041504 (376h51m7.97633619s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050418785 (376h51m7.976713485s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.050860042 (376h51m7.977154734s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051563500 (376h51m7.977858192s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.051963243 (376h51m7.978257933s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052302595 (376h51m7.978597349s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.052672504 (376h51m7.978967197s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053100417 (376h51m7.979395114s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053445269 (376h51m7.979739955s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.053883256 (376h51m7.980177948s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.054279476 (376h51m7.980574162s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Uploading piece... Jul 9 02:02:41.130875320 (376h51m8.057170014s) [upload=301809]: Next returned nil segment Jul 9 02:02:41.130895759 (376h51m8.057190457s) [upload=301809]: Waiting for error group managing segments... Jul 9 02:02:41.136204286 (376h51m8.062498966s) [noise=true , address=77.90.31.12:4590 , piece_id=PLFCPFXCWG3NZD36E2H63RTG3AB7OQW2ZF7MFONJAH5EMTCHBOEA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.136645242 (376h51m8.062939934s) [noise=true , address=45.91.116.6:4975 , piece_id=CWDV3VFRX35XKSSOZSB3PKQZLCHUJ7JQYSSRPBPIFCXNORINCF7Q, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137355020 (376h51m8.063649707s) [noise=true , address=45.91.116.11:4118 , piece_id=HWIPPK75SWQ6WZXZUFII76UVHSOSQLYHFT564KBE4QK2VXXLK6EQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.137902951 (376h51m8.064197633s) [noise=true , address=77.90.31.13:4627 , piece_id=EL7YUPQYPWFJXRCNVQOPCLNFTWC3XSFOGNMYAKX4K6BZ46G5LLMA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139285699 (376h51m8.065580372s) [noise=true , address=45.91.116.7:4009 , piece_id=CZ43TVEQXV5A7QKWZ3JCB52VXJK3R6YEZ52NF3LWJXLLBEJNTA6A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.139410254 (376h51m8.06570494s) [noise=true , address=45.91.116.5:4297 , piece_id=T4NZOKWJC5ZKZSZUPPF3JGGTNCK24ODP5ICLGXZRYJ6WXHSHRL7A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141152669 (376h51m8.067447346s) [noise=true , address=104.36.82.16:4295 , piece_id=U4I22LCJMXOI6KMC2YPRNOOX3DE7SY2G5YW6GG4X4AMILSME7PNA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.141684412 (376h51m8.067979085s) [noise=true , address=45.91.116.10:4081 , piece_id=QGXAPD4MQZRI5X2CGBNCYPLQXEH7W2BXPWNERFQXPCXDJ25QKULA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.142262736 (376h51m8.068557427s) [noise=true , address=45.91.116.9:4263 , piece_id=NOAT5GFPNMEYRZJYR25PE3SZ5NUP3ZOUQYUK2Y7EXZ6XHS2SV7OQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:41.145949224 (376h51m8.072243912s) [noise=true , address=104.36.82.9:4006 , piece_id=QHC5FRUKVWL5M6TYSQWP2JX3FGZDSIYHPPF2Y6L5OXZEKTLKDOUQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:02:56.644337945 (376h51m23.570632617s) [noise=true , address=104.36.82.10:4077 , piece_id=32KKYDXSU2QO6HXIIN2YAOFKJ7BPM7XHHVWNNHRAAIFK6Y3MICEQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:00.255623941 (376h51m27.18191863s) [noise=true , address=104.36.82.11:4115 , piece_id=RFMJCCKX6Y5BU33MFUTHT7L2PW4ST5IHI7XH33DQQCCNLS3VYCDQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: <nil> Jul 9 02:03:10.988827693 (376h51m37.915122369s) [noise=true , address=104.36.82.12:4149 , piece_id=ESRHGGJFNGQI5G22IPM6EQKHCEFARMLE2DWJAMA5LGVFB3WKH3XQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988897302 (376h51m37.915191981s) [noise=true , address=104.36.82.2:4732 , piece_id=D5DPHAMK7VCC7DRJYMCAKCLN7I6PLAV7X6C2TWELJLXJTOQY3TZQ, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.988913335 (376h51m37.915208027s) [noise=true , address=104.36.82.15:4259 , piece_id=GQU6XAU2DWXRN27FWRIJWB3BITKDO5B6BGELN2JRAXHSWJCMXOKA, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:03:10.989022369 (376h51m37.915317066s) [noise=true , address=104.36.82.14:4222 , piece_id=SEEFCXBQJQQHQT7LZ3APEV26YF4DAY2BQMR3X3YSOVEYIFYV242A, seg_pos={0 0}, upload=301809]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:32788->104.36.82.14:4222: use of closed network connection Jul 9 02:03:10.989097572 (376h51m37.915392265s) [seg_pos={0 0}, upload=301809]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:03:11.020835208 (376h51m37.947129912s) [upload=301809]: Done uploading segments... Jul 9 02:03:11.020880111 (376h51m37.9471748s) [upload=301809]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 619.026114ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..X....X.X.X......X......X...............X....X....X......] duration: 18.294753051s FAIL (2025-07-09 02:03:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.294836202s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [.X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX] duration: 1m0.003078489s FAIL (2025-07-09 02:03:00 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00309381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..X....X.X.X......X......X...............X....X....X.....] duration: 9.211711379s FAIL (2025-07-09 02:02:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.2117704s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [..X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....X] duration: 1m0.001037413s FAIL (2025-07-09 02:02:00 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00105994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 489.492919ms FAIL (2025-07-09 02:01:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 489.520699ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..X....X.X.X......X......X...............X....X....X....] duration: 17.108496229s FAIL (2025-07-09 02:01:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.108524464s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 451.625972ms FAIL (2025-07-09 02:01:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.656184ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..X....X.X.X......X......X...............X....X....X...] duration: 25.03756002s FAIL (2025-07-09 02:01:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.037610378s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [...X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....] duration: 48.60199139s FAIL (2025-07-09 02:00:56 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.602015889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 649.382858ms FAIL (2025-07-09 02:00:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 649.433154ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [....X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX...] duration: 27.254948152s FAIL (2025-07-09 02:00:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.254962128s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..X....X.X.X......X......X...............X....X....X..] duration: 14.317248811s FAIL (2025-07-09 02:00:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.317304266s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 547.676246ms FAIL (2025-07-09 01:59:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 547.728048ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X..X....X.X.X......X......X...............X....X....X.] duration: 27.135472193s FAIL (2025-07-09 01:59:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.135516631s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [.....X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX..] duration: 58.565584998s FAIL (2025-07-09 01:59:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 58.565790944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 382.425065ms FAIL (2025-07-09 01:59:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 382.4538ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X..X....X.X.X......X......X...............X....X....X] duration: 30.000712046s FAIL (2025-07-09 01:58:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000733059s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 20.571226343s FAIL (2025-07-09 01:58:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 20.57126087s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX.] duration: 9.35517614s FAIL (2025-07-09 01:58:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.355189316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..XX] duration: 1m0.000675488s FAIL (2025-07-09 01:58:13 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000691789s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)