Orbiter History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 33.989994ms target: https://us1.storj.io/login FAIL (2025-06-26 10:27:50 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 34.014324ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 98.169442ms target: https://us1.storj.io/login FAIL (2025-06-26 10:27:29 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 98.210006ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 64.249988ms target: https://us1.storj.io/login FAIL (2025-06-26 10:26:45 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 64.278636ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 32.015204ms target: https://us1.storj.io/login FAIL (2025-06-26 10:25:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 32.038386ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 88.436891ms target: https://us1.storj.io/login FAIL (2025-06-26 10:24:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 88.456148ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 128.941769ms target: https://us1.storj.io/login FAIL (2025-06-26 10:24:03 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 128.980608ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 30.118557ms target: https://us1.storj.io/login FAIL (2025-06-26 10:23:38 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 30.14846ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 31.293778ms target: https://us1.storj.io/login FAIL (2025-06-26 10:22:51 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 31.317627ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 205.640733ms target: https://us1.storj.io/login FAIL (2025-06-26 10:22:07 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 205.677021ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 215.610956ms target: https://us1.storj.io/login FAIL (2025-06-26 10:21:20 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 215.655497ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 151.945333ms target: https://us1.storj.io/login FAIL (2025-06-26 10:20:38 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 151.971445ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 33.28033ms target: https://us1.storj.io/login FAIL (2025-06-26 10:20:00 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 33.307848ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 28.976532ms target: https://us1.storj.io/login FAIL (2025-06-26 10:19:42 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 28.99939ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 28.064903ms target: https://us1.storj.io/login FAIL (2025-06-26 10:18:42 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 28.085042ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 97.49223ms target: https://us1.storj.io/login FAIL (2025-06-26 10:17:32 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 97.528247ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.574313613s target: https://us1.storj.io/login FAIL (2025-06-26 10:17:08 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 1.574338306s core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 32.480761ms target: https://us1.storj.io/login FAIL (2025-06-26 10:16:07 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 32.500622ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 38.902907ms target: https://us1.storj.io/login FAIL (2025-06-26 10:14:56 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 38.946378ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 28.97417ms target: https://us1.storj.io/login FAIL (2025-06-26 10:14:07 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 29.015466ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 27.482633ms target: https://us1.storj.io/login FAIL (2025-06-26 10:13:13 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 27.509058ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 228.398836ms target: https://us1.storj.io/login FAIL (2025-06-26 10:12:29 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 228.424721ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 37.570322ms target: https://us1.storj.io/login FAIL (2025-06-26 10:11:32 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 37.603962ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 32.618709ms target: https://us1.storj.io/login FAIL (2025-06-26 10:10:50 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 32.64219ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 31.764856ms target: https://us1.storj.io/login FAIL (2025-06-26 10:10:25 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 31.79054ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 48.046714ms target: https://us1.storj.io/login FAIL (2025-06-26 10:10:03 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 48.077001ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 31.559015ms target: https://us1.storj.io/login FAIL (2025-06-26 10:09:25 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 31.587376ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 276.178758ms target: https://us1.storj.io/login FAIL (2025-06-26 10:08:16 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 276.210819ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 26.481298ms target: https://us1.storj.io/login FAIL (2025-06-26 10:07:13 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 26.573867ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 55.091663ms target: https://us1.storj.io/login FAIL (2025-06-26 10:06:04 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 55.120571ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 289.999365ms target: https://us1.storj.io/login FAIL (2025-06-26 10:05:01 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 290.025249ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 125.932348ms target: https://us1.storj.io/login FAIL (2025-06-26 10:04:16 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 125.964906ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 29.092145ms target: https://us1.storj.io/login FAIL (2025-06-26 10:03:08 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 29.114481ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001221348s target: https://us1.storj.io/login FAIL (2025-06-26 10:03:07 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: i/o timeout Duration: 30.00125217s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 417.727535ms FAIL (2025-06-26 06:53:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 417.756879ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 396.176759ms FAIL (2025-06-26 06:52:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 396.201455ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 562.933467ms FAIL (2025-06-26 06:52:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 562.961025ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 465.406849ms FAIL (2025-06-26 06:52:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 465.459683ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 494.034035ms FAIL (2025-06-26 06:51:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.058778ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 525.915498ms FAIL (2025-06-26 06:51:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.98924ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 472.448763ms FAIL (2025-06-26 06:51:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 472.477271ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 549.021842ms FAIL (2025-06-26 06:50:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 549.072058ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 447.657341ms FAIL (2025-06-26 06:50:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 447.69027ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 408.395974ms FAIL (2025-06-26 06:49:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 408.489039ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 651.850819ms FAIL (2025-06-26 06:49:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 651.923228ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 455.054884ms FAIL (2025-06-26 06:49:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 455.083853ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 413.629032ms FAIL (2025-06-26 06:49:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 413.674916ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 510.420211ms FAIL (2025-06-26 06:48:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 510.474542ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 503.883226ms FAIL (2025-06-26 06:48:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 503.948355ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 453.38883ms FAIL (2025-06-26 06:48:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 453.44586ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 512.681996ms FAIL (2025-06-26 06:48:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 512.711067ms edge gateway us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)