Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [.X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX..] duration: 35.635363899s FAIL (2025-07-09 01:56:13 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.635977163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 464.115487ms FAIL (2025-07-09 01:55:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 464.138744ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX........X..X....X.X.X......X......X...............X....X.] duration: 13.434939365s FAIL (2025-07-09 01:55:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.43496881s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX.] duration: 37.290032656s FAIL (2025-07-09 01:55:24 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.290226637s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXX........X..X....X.X.X......X......X...............X....X] duration: 30.000895151s FAIL (2025-07-09 01:55:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000918495s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 491.958674ms FAIL (2025-07-09 01:54:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.984292ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [X..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXXX] duration: 1m0.000694581s FAIL (2025-07-09 01:54:47 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000707524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..XXX........X..X....X.X.X......X......X...............X....] duration: 25.207013776s FAIL (2025-07-09 01:54:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.207031759s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXXX] duration: 1m0.001697786s FAIL (2025-07-09 01:53:47 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00172218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 554.007536ms FAIL (2025-07-09 01:53:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 554.05248ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...XXX........X..X....X.X.X......X......X...............X...] duration: 501.855146ms FAIL (2025-07-09 01:53:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 501.880555ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 448.006837ms FAIL (2025-07-09 01:52:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 448.027428ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [.XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXXX] duration: 1m0.00139087s FAIL (2025-07-09 01:52:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001405415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 451.622895ms FAIL (2025-07-09 01:52:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.648323ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....XXX........X..X....X.X.X......X......X...............X..] duration: 961.477341ms FAIL (2025-07-09 01:52:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 961.503242ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [..XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXXX] duration: 1m0.001163271s FAIL (2025-07-09 01:51:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001194694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....XXX........X..X....X.X.X......X......X...............X.] duration: 14.031081175s FAIL (2025-07-09 01:51:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.031104299s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 390.987232ms FAIL (2025-07-09 01:51:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 391.019768ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......XXX........X..X....X.X.X......X......X...............X] duration: 30.001045902s FAIL (2025-07-09 01:51:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001071452s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 429.322685ms FAIL (2025-07-09 01:50:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.375595ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [...XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XXX] duration: 1m0.000704806s FAIL (2025-07-09 01:50:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000717882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 419.039022ms FAIL (2025-07-09 01:49:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.069464ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......XXX........X..X....X.X.X......X......X...............] duration: 1.653742266s FAIL (2025-07-09 01:49:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.653769975s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........XXX........X..X....X.X.X......X......X..............] duration: 24.82891682s FAIL (2025-07-09 01:49:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.828945566s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.XX] duration: 1m0.000561202s FAIL (2025-07-09 01:49:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000576798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 478.427697ms FAIL (2025-07-09 01:49:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.457976ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [X....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.X] duration: 1m0.001093459s FAIL (2025-07-09 01:48:37 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001112596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 407.116831ms FAIL (2025-07-09 01:48:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.147927ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 439.23185ms FAIL (2025-07-09 01:48:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.265123ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........XXX........X..X....X.X.X......X......X.............] duration: 819.556256ms FAIL (2025-07-09 01:48:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 820.335624ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 458.139301ms FAIL (2025-07-09 01:47:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.169864ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [.X....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X.] duration: 44.272245223s FAIL (2025-07-09 01:47:37 UTC) context deadline exceeded Duration: 44.272261366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 22.91958845s FAIL (2025-07-09 01:47:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 22.919623105s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.........XXX........X..X....X.X.X......X......X............] duration: 3.954460294s FAIL (2025-07-09 01:46:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.954495453s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [..X....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.X] duration: 1m0.000356559s FAIL (2025-07-09 01:46:52 UTC) context deadline exceeded Duration: 1m0.000382043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 470.446583ms FAIL (2025-07-09 01:46:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.478108ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.........XXX........X..X....X.X.X......X......X...........] duration: 19.311379008s FAIL (2025-07-09 01:46:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.311410477s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [X..X....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X.] duration: 46.228521201s FAIL (2025-07-09 01:45:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.228541984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 427.762259ms FAIL (2025-07-09 01:45:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.796589ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.........XXX........X..X....X.X.X......X......X..........] duration: 22.60867956s FAIL (2025-07-09 01:45:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.608705293s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [XX..X....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.X] duration: 1m0.000774518s FAIL (2025-07-09 01:45:06 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000796188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 451.069062ms FAIL (2025-07-09 01:44:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.12102ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.........XXX........X..X....X.X.X......X......X.........] duration: 26.570662152s FAIL (2025-07-09 01:44:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.570684301s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 403.777081ms FAIL (2025-07-09 01:44:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.802049ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.........XXX........X..X....X.X.X......X......X........] duration: 17.196335178s FAIL (2025-07-09 01:43:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.196350877s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [.XX..X....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX.] duration: 33.448819049s FAIL (2025-07-09 01:43:35 UTC) context deadline exceeded Duration: 33.448842739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.........XXX........X..X....X.X.X......X......X.......] duration: 19.135422003s FAIL (2025-07-09 01:43:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.13544897s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 411.475147ms FAIL (2025-07-09 01:43:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.502531ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.02836282s FAIL (2025-07-09 01:42:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jul 9 01:41:50.662987419 (376h30m17.589282106s) [seg_pos={0 0}, upload=301526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 01:41:50.663080661 (376h30m17.589375347s) [upload=301526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 01:41:50.663085482 (376h30m17.589380171s) [upload=301526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 01:41:50.725362493 (376h30m17.651657186s) [upload=301526]: Done uploading segments... Jul 9 01:41:50.725384461 (376h30m17.651679154s) [upload=301526]: Upload finished. err: <nil> Jul 9 01:42:20.034751550 (376h30m46.961046227s) [upload=301531]: Starting upload Jul 9 01:42:20.034818898 (376h30m46.961113576s) [upload=301531]: Done starting upload Jul 9 01:42:20.035052585 (376h30m46.96134726s) [upload=301531]: Uploading segments... Jul 9 01:42:20.035120907 (376h30m46.961415602s) [upload=301531]: Got next segment. Inline: false Jul 9 01:42:20.035156051 (376h30m46.961450724s) [upload=301531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 01:42:20.035163250 (376h30m46.961457936s) [upload=301531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 01:42:20.071677462 (376h30m46.997972159s) [seg_pos={0 0}, upload=301531]: Begin upload segment... Jul 9 01:42:20.071763990 (376h30m46.998058684s) [seg_pos={0 0}, upload=301531]: Done begin upload segment. Jul 9 01:42:20.072001144 (376h30m46.998295825s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072178564 (376h30m46.998473249s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.072750022 (376h30m46.999044714s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074042928 (376h30m47.000337612s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.074799406 (376h30m47.00109409s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.075695171 (376h30m47.001989851s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076153086 (376h30m47.002447765s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.076759091 (376h30m47.00305377s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077239192 (376h30m47.003533871s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.077865192 (376h30m47.004159877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.078494032 (376h30m47.004788713s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079272408 (376h30m47.005567088s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.079900203 (376h30m47.006194881s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.080482924 (376h30m47.006777603s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.082435052 (376h30m47.00872973s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.084332485 (376h30m47.010627166s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Uploading piece... Jul 9 01:42:20.165816048 (376h30m47.092110726s) [upload=301531]: Next returned nil segment Jul 9 01:42:20.165834674 (376h30m47.092129371s) [upload=301531]: Waiting for error group managing segments... Jul 9 01:42:20.170947359 (376h30m47.097242039s) [noise=true , address=45.91.116.14:4191 , piece_id=WRPB3DLJSNHOWCH6WIRTKVGFK7NS6KZCOMS7LVUBLR7NPFXSYXTA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171492316 (376h30m47.097786992s) [noise=true , address=45.91.116.6:4975 , piece_id=XNRLCUEXYOJYM4MTARCZCNEI7ZREW7L5UOEPJK3NZRAPZ36OBWVA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.171883300 (376h30m47.09817798s) [noise=true , address=45.91.116.6:4973 , piece_id=5J6I2UA7QWRSVZX6FIW63JPNSK6VUN2ASCUG6LZGRBNZGZD6CKJQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172291210 (376h30m47.098585889s) [noise=true , address=77.90.31.14:4662 , piece_id=YXOKBZWT5BBVVOXUO75SVOIW5GZGBUK2TVSZDVFAXSOXNE7Z3GSQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.172667567 (376h30m47.098962247s) [noise=true , address=45.91.116.11:4118 , piece_id=TFEM67YAQ2NOM3ABV5BN7WNVMTVKM5Z7IKTYWQBDSEIAIUT6NKQQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173016899 (376h30m47.099311577s) [noise=true , address=45.91.116.4:4939 , piece_id=VXW5SXCKZZD6QHWRV36TKB4WXFMBMHYLHAL5AUYTPRARFXHVOUXQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.173472378 (376h30m47.099767055s) [noise=true , address=45.91.116.7:4010 , piece_id=4KK66ANKFNKSUC5LGMHUMRFRXJB6KVFKTCHFQI5ABXQDXTBPQOBQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.174049990 (376h30m47.100344678s) [noise=true , address=45.91.116.10:4081 , piece_id=CVI45ZKNEPPSCOQTDDY4WHHRKEG7NAH7IEQ4DJUKI3NFW2YIEIYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.180174079 (376h30m47.106468765s) [noise=true , address=77.90.31.15:4699 , piece_id=EJT2SJJOYFDGD63BH2DWMAOPRPMY65PPTLXI7I4PW4CPMNAJFBGQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:20.188382392 (376h30m47.114677065s) [noise=true , address=77.90.31.17:4772 , piece_id=EUSNASJNVKOYSYXJ37VY5C4QQZQ3JQJ5ILXB62JSWKMYMPUTKHYA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:36.255050243 (376h31m3.181344922s) [noise=true , address=104.36.82.2:4731 , piece_id=QNPZ7LQLCFAN5BQSBKMPN2ANTQEQ76JLMVJ24NTA7EYOVHDTPDNQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:41.197120332 (376h31m8.123415024s) [noise=true , address=104.36.82.15:4258 , piece_id=FGQXQBH3KOFXGZMHKWJ2MPR5SBL2FZB3IVXM4SZTT6BZUJ3EOQTQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: <nil> Jul 9 01:42:50.003796687 (376h31m16.930091369s) [noise=true , address=104.36.82.11:4114 , piece_id=YWPNVIYGKTD7WJKOLBD4QV5D3O5FKFSXX3NICCEKAZADY6B7FPEQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.003887380 (376h31m16.930182066s) [noise=true , address=104.36.82.5:4837 , piece_id=WYTLW3JSMUM3SKTS7GYRXVUBGCH6YQZXZ7CYHJAYXDZAPBWEAXXA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 01:42:50.004158031 (376h31m16.930452718s) [noise=true , address=104.36.82.14:4222 , piece_id=QHIDV4JM4ZHN5VYBIVTT3VT4YJ75UYA3PQCREKB5YP5627MYNOCA, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37022->104.36.82.14:4222: use of closed network connection Jul 9 01:42:50.004278701 (376h31m16.930573387s) [noise=true , address=104.36.82.4:4802 , piece_id=RDRQVIAL2K7N3QAKHL3OZNPTA4IRJR64WZMFNLYVQXNNK3NCM7KQ, seg_pos={0 0}, upload=301531]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57884->104.36.82.4:4802: use of closed network connection Jul 9 01:42:50.004302554 (376h31m16.93059724s) [seg_pos={0 0}, upload=301531]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 01:42:50.029482661 (376h31m16.955777352s) [upload=301531]: Done uploading segments... Jul 9 01:42:50.029543155 (376h31m16.955837828s) [upload=301531]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028388363s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [..XX..X....XX..X.......X....XX..X..XX..XX.X..XX...XX.XXX.XXX] duration: 1m0.000735034s FAIL (2025-07-09 01:42:48 UTC) context deadline exceeded Duration: 1m0.000750448s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)