Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..........................X.] duration: 2.67027674s trace: https://jaeger.dev.storj.io/trace/35466b767af05c02 FAIL (2025-12-07 13:31:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.670311307s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..........................X] duration: 30.373705819s trace: https://jaeger.dev.storj.io/trace/35466b767af05c02 FAIL (2025-12-07 13:31:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.373742567s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.690849936s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:30:15 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.690876932s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.604254229s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:29:07 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.604284778s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.629365243s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:28:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.629388962s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.586686451s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:27:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.586722627s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.624171183s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:27:33 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.62418547s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.669866384s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:26:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.669893933s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.579369031s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:25:48 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.579395764s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.563795499s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:24:52 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.563821111s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 3.636691973s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:24:11 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 3.636719633s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.648387005s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:23:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.648414544s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.694010608s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:23:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.694039695s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.591972127s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:21:59 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.591999227s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.628426376s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:21:14 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.62845118s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.437353805s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:20:46 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.437379367s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.502859608s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:19:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.502884816s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.564041235s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:18:50 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.564069199s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.581144843s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:17:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.58129549s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.609457661s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:17:10 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.609505316s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.549438458s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:16:41 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.549464703s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.547868488s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:16:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.547897856s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.604406306s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:15:39 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.604435976s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.604746501s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:15:23 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.604770277s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.678354567s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:14:14 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.678380427s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.65655833s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:13:33 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.656584738s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.743024581s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:13:08 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.743072427s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 9.83430721s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:12:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 9.834322117s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002325092s trace: https://jaeger.dev.storj.io/trace/436d5a56a7dc734 FAIL (2025-12-07 13:12:03 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.002417429s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 384.683078ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:31:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 384.711108ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 403.560874ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:30:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.588776ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 361.857953ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:29:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 361.888544ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 429.576576ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:29:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.600662ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 412.311749ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:28:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.335471ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 516.382772ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:27:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 516.406014ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 409.14902ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:26:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 409.172208ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 376.073095ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:25:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 376.100106ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 383.735331ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:25:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.772997ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 387.382974ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:24:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.406294ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 381.190386ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:23:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 381.214384ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 479.179681ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:22:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.206303ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 441.905931ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:21:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.932912ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 494.967663ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.991375ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 383.90119ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:20:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.931357ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 540.675103ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:20:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 540.774238ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 367.587362ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:19:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.611338ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 415.181159ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:18:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.209471ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 386.262183ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 386.294434ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 473.562394ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.595921ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 392.90114ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.927144ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)