Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
457.351217ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:55:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 457.373866ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
434.634711ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:54:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 434.716817ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
467.445947ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:53:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 467.476325ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
519.679529ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:53:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 519.777669ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
447.636064ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:52:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 447.653484ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
458.126169ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:51:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 458.187532ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
494.851223ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:51:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 494.894558ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
463.287069ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:50:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.31656ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
467.458713ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:49:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 467.476983ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
470.377868ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:48:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 470.409531ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
501.559609ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:48:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.585594ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
483.162201ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:47:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 483.226094ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
452.093674ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:46:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 452.342297ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
447.625512ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:45:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 447.856719ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
478.726519ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:45:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 478.759561ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
477.833988ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:43:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.883236ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
470.044813ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:43:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 470.071899ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
435.022116ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:42:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 435.05043ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
441.96189ms trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:42:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 441.98446ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000815778s trace:
https://jaeger.dev.storj.io/trace/72da01ee55825f29
FAIL (2025-12-09 06:42:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000827806s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.352527893s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:23:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 5.352557978s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
3.320509079s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:23:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.320549618s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
8.997766068s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:22:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 8.99779282s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
3.373433896s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:21:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.373461511s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
16.324013845s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:21:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 16.324051829s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.772198324s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:20:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 6.772266893s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.175314819s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:19:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 6.175338032s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
3.564381885s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:18:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.564408476s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
4.938610261s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:17:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 4.93865401s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
7.449529491s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:17:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 7.449556882s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
3.643802188s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:16:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.643819077s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
4.005283508s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:16:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 4.005351089s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
11.194941553s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:16:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 11.195013932s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
4.054157773s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:15:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 4.05418539s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.976889654s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:14:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 6.976916704s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
3.074876245s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:14:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.075012574s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
7.142993103s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:13:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 7.143034842s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
3.340232619s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:12:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.340324161s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
7.299137512s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:11:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 7.299163003s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.467074097s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:11:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 5.467100248s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
9.768544477s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:11:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 9.768573287s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
6.825213945s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:10:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 6.825248774s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
10.255576315s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:09:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 10.255601867s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
3.793810409s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:09:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.793858922s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
3.230207365s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:08:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.230244118s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
3.315937496s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:07:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 3.315965332s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.821912645s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:07:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 2.821940401s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
4.419024278s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:06:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 4.419108335s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
7.523619124s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:06:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 7.523661358s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
6.991568835s trace:
https://jaeger.dev.storj.io/trace/1ae48fde171f1e73
FAIL (2025-12-09 01:05:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
431]: Uploading piece...
Dec 9 00:42:18.349555508 (81h14m19.177263902s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.349915214 (81h14m19.177623605s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350255805 (81h14m19.177964202s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350420977 (81h14m19.178129371s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.350744803 (81h14m19.178453197s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351159312 (81h14m19.178867703s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351523348 (81h14m19.17923174s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351536441 (81h14m19.179244835s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.351898017 (81h14m19.179606411s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.352443120 (81h14m19.180151515s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Uploading piece...
Dec 9 00:42:18.787967509 (81h14m19.6156759s) [upload=84431]: Next returned nil segment
Dec 9 00:42:18.788012770 (81h14m19.615721163s) [upload=84431]: Waiting for error group managing segments...
Dec 9 00:42:18.957843773 (81h14m19.785552172s) [noise=true , address=81.171.14.70:4025 , piece_id=WKCLO53IZOQJJBEWAOFWA6BVPZZGZ7BR2C2Y7WSC3TYRHNAHNVIQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.964776538 (81h14m19.79248493s) [noise=true , address=81.171.14.70:4030 , piece_id=UHQEPVT2EDFXBLAEXRXHTOJ4RX2TG62HZUWC7BSKPSC7XHLKFB4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:18.993346643 (81h14m19.821055025s) [noise=true , address=81.171.14.70:4036 , piece_id=25PA6KBT63FB5S6ITLFLYDAL4434TCCANMFKB245BE5QV7QQVTRQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.012142948 (81h14m19.839851351s) [noise=true , address=81.171.14.70:4041 , piece_id=2VQG2RNJDGSJXILJO3OJ4U7E7FPRFJPKILJTKFYOYDWBSVD2RFRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.022826121 (81h14m19.850534519s) [noise=true , address=81.171.14.70:4033 , piece_id=ARCKWP3MNFEPBMTXLVHZX4LFPGGTHONTSOASUNGCS5BQUXVJG4HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.063578853 (81h14m19.891287244s) [noise=true , address=89.149.201.147:4010 , piece_id=TLVJNVTD5BQWQTZQIWJYAWPXVCMT3IMEMOB7V7Q6ZNSN5626Z4CQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.342882228 (81h14m20.170590621s) [noise=true , address=89.149.201.147:4022 , piece_id=CO2G6Y7QHLUMGFRDVAV54RKKVAOR62VTXKTHW7WFDGWRCRQDGKXQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.375538173 (81h14m20.203246554s) [noise=true , address=185.167.204.206:4038 , piece_id=3VP7QJ2UMTISCSV76HZTV5XEQ7ENEHKFLUNP7JQX6VTM53W57ADA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.413795687 (81h14m20.241504094s) [noise=true , address=81.171.14.70:4031 , piece_id=2EZH3GEHB4DU6NVDAVXDSO34CAKP4JA6JNDIUOUUHIDN4ARTKNVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.447694798 (81h14m20.275403199s) [noise=true , address=185.167.204.206:4048 , piece_id=S4XZ7VAPSKHBDHRMS67GDZJWCOMAFLMGMEYJA3Y5SM5XHT7FYJRA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.458660706 (81h14m20.286369097s) [noise=true , address=81.171.14.70:4040 , piece_id=3T6KZGJPMGIUHAIVCGYE3L6MCORLWG5YJT22XVXSAVQ6VKQBWAXA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.459487553 (81h14m20.287195945s) [noise=true , address=185.167.204.205:4007 , piece_id=4NFS3Y4ZKSYDEDOMIQQ62CUSWKZPGP43NDIEBV2IT2ZVDJLD744A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.493534151 (81h14m20.321242543s) [noise=true , address=185.167.204.206:4070 , piece_id=B6BRUJUF3EQQXH7SZE4ZHLSTQNOLINCO7NCE6SJOT555XDOQER5Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.534850646 (81h14m20.362559037s) [noise=true , address=185.167.204.205:4033 , piece_id=RTPDKSUKCZW57ZMMLMVGSQUZPZKDYXOBDEZCJME4DVZZIRZ733UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.559286441 (81h14m20.386994832s) [noise=true , address=89.149.201.147:4015 , piece_id=VDRCP7KNVW3ZYE5SZ5SNMCAM7FHF4CFGLWRQEW5GCENB4RYP57UQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.596129224 (81h14m20.423837615s) [noise=true , address=89.149.201.147:4023 , piece_id=ROEJ5V7NXGCY67YDGAFKTWZJDHBADI3GW77VRW42FVLBMMBLXOQA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.651975496 (81h14m20.479683887s) [noise=true , address=185.167.204.206:4039 , piece_id=DPKTXVTDHTKY6VB3M3VT7FLOWIABR7VYA45LS7EJSPMCPZ6B26UA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.831424466 (81h14m20.659132857s) [noise=true , address=185.167.204.206:4058 , piece_id=DK5I65SFZWRLTIPH6FY3QGKLINPLIIYW7P4RXT47RAGZECHPMV4A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.835500067 (81h14m20.663208461s) [noise=true , address=185.167.204.205:4018 , piece_id=KVSE7SHZZNXSBZ6TJIMXKYBPKZGJHJJM6JYA6FCSYLXPUWX7N7GQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.859903205 (81h14m20.687611596s) [noise=true , address=185.167.204.206:4064 , piece_id=2L4P35K7UTAIV7PLEPABCIZSYOENQ37LA5VS6AMQNNBRMOVMLU3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:19.986818261 (81h14m20.814526659s) [noise=true , address=89.149.201.147:4004 , piece_id=JVKBZJ33GL3I6EAHVGZWROKXV6UHE4WENFABWMLQ7OTGMDL3ODEA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.393469827 (81h14m21.221178213s) [noise=true , address=89.149.201.147:4014 , piece_id=N5VVAVJKLHCMIT2LOFVLUZBF6TOKOVSI6O23JEOS6R6UVY45EX4Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.741878447 (81h14m21.569586842s) [noise=true , address=89.149.201.147:4024 , piece_id=2US4JMMMZPJ5FNIQMJHDTEENMI4KC5AXYWUP3XQA57Y3G3E2SI6A, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:20.840486734 (81h14m21.668195128s) [noise=true , address=185.167.204.206:4068 , piece_id=SRZHDHCECW7B67OZF2DOZVGN25XJC7BUT6AOJJUAQENB7OI5SVVA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.527454216 (81h14m22.355162608s) [noise=true , address=185.167.204.206:4053 , piece_id=FCMLI7FHQPRU4PZCMD4KNNUCGALNLJTWSAXO4CBESISOUG3VA2HQ, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633923496 (81h14m22.461631899s) [noise=true , address=185.167.204.206:4072 , piece_id=RQ4WVVOZV5J4PQE2EVMMN3CULDE4UB5G444X3S2IK3CJJIARRC3Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: <nil>
Dec 9 00:42:21.633950916 (81h14m22.461659317s) [seg_pos={0 0}, upload=84431]: Segment reached optimal threshold of 26 pieces.
Dec 9 00:42:21.634329611 (81h14m22.462038002s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF): protocol: expected piece hash; context canceled; EOF
Dec 9 00:42:21.634342626 (81h14m22.462051023s) [noise=true , address=185.167.204.205:4003 , piece_id=GJNZ2CFCNEJH4ROMH4FL7ANYX5FOHFFMZDI2QYFKQMUWBN5LTK7Q, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634383757 (81h14m22.462092159s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg): protocol: read tcp 159.223.143.39:44726->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 00:42:21.634400552 (81h14m22.462108944s) [noise=true , address=185.167.204.206:4066 , piece_id=LMF5QLN5FSB6X2MIUTEK6PGN65MKZFKVEYQ36GDULJ7DDFQLBDOA, seg_pos={0 0}, upload=84431]: Long tail cancellation due to optimal threshold
Dec 9 00:42:21.634423057 (81h14m22.462131435s) [seg_pos={0 0}, upload=84431]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 00:42:21.634528085 (81h14m22.462236483s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 00:42:21.634535147 (81h14m22.462243546s) [upload=84431]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 00:42:21.793166703 (81h14m22.620875098s) [upload=84431]: Done uploading segments...
Dec 9 00:42:21.793210854 (81h14m22.620919248s) [upload=84431]: Upload finished. err: <nil>
Duration: 6.991584241s
core
uplink
eu-select-1
eu1-placement-15
|