Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXX....................] duration:
106.211973ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:55:41 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 106.237027ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXX...................] duration:
114.538794ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:54:39 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 114.55798ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXX..................] duration:
103.80417ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:54:19 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 103.828085ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXX.................] duration:
112.509026ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:53:57 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 112.531984ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXX................] duration:
111.939704ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:53:24 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 111.967127ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXX...............] duration:
109.808361ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:52:48 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 109.829991ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXX..............] duration:
111.670879ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:52:28 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 111.698904ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXX.............] duration:
111.065146ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:51:15 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 111.122014ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXX............] duration:
115.933043ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:50:41 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 115.957855ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXX...........] duration:
113.03675ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:50:02 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 113.081016ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXX..........] duration:
113.144243ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:48:53 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 113.168595ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXX.........] duration:
110.509541ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:47:47 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 110.552352ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXX........] duration:
110.611559ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:46:48 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 110.633655ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXX.......] duration:
119.799153ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:45:33 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 119.82265ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXX......] duration:
132.083447ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:45:02 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 132.147884ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXX.....] duration:
110.485039ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:44:07 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 110.531104ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXX....] duration:
119.711473ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:43:31 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 119.791478ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXX...] duration:
115.058459ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:43:09 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 115.084835ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXX..] duration:
110.377122ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:42:48 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 110.400715ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXX.] duration:
119.335097ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:41:49 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 119.366165ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration:
36.729323ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/24858b5668cafa5c
FAIL (2026-03-12 02:40:35 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 36.751402ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration:
57.004967ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/7301d4910009c332
FAIL (2026-03-12 02:39:51 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 57.032583ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration:
61.757469ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/265e8aaf233ce52d
FAIL (2026-03-12 02:38:43 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 61.769155ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration:
36.357741ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/273bf9fec90370b4
FAIL (2026-03-12 02:38:12 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 36.385086ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration:
36.886821ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/3d76a8438587bbe4
FAIL (2026-03-12 02:37:06 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 36.969231ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration:
34.006362ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/53c89216e05641f7
FAIL (2026-03-12 02:36:13 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 34.03996ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
39.628003ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/63859cb5c13016a9
FAIL (2026-03-12 02:35:26 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 39.662677ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
36.798734ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/39a672754d2d62a1
FAIL (2026-03-12 02:34:41 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 36.823911ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
37.422431ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/3dedb81fb51052bd
FAIL (2026-03-12 02:34:24 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 37.435474ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration:
37.135445ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/356d2f54a695fc49
FAIL (2026-03-12 02:33:35 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 37.159785ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
4 out of 60 is failed (threshold=3) [........................................................XXXX] duration:
34.508771ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/152c3db97d22fa3a
FAIL (2026-03-12 02:33:06 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 34.539759ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
39.105392ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/7fcce31e7a177a55
FAIL (2026-03-12 02:32:10 UTC)
Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 39.131113ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
37.409271ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/44dd31b49bc9d76c
FAIL (2026-03-12 02:31:49 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 37.420634ms
core
website
|
checkers.HTTP{"GET" "https://version.storj.io/"} failure (antiblip window 3/60): (http.version_storj_io)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
36.607768ms target:
https://version.storj.io/ trace:
https://jaeger.dev.storj.io/trace/2f5f7e2c09c32bc0
FAIL (2026-03-12 02:31:24 UTC)
orbiter warning: Get "https://version.storj.io/": dial tcp 34.173.164.90:443: connect: connection refused
Duration: 36.633963ms
core
website
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.01410098s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:23:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 2.014128825s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.899329371s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:23:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.899354393s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.944434736s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:22:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.94446295s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.001221334s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:21:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 2.00127982s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.956523521s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:19:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.956549466s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.970523124s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:18:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.970555555s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.994500034s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:18:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.994599673s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.983937759s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:16:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.983969094s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.993359902s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:16:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.993401051s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.245177886s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:15:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 2.245207433s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
1.99671538s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:13:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.996743659s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
1.861223329s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:12:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.861265943s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
2.030165841s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:12:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 2.030260366s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
1.903721342s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:11:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.903747441s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
1.8373851s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:10:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.837448825s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
1.976811183s trace:
https://jaeger.dev.storj.io/trace/116fb63136b7258c
FAIL (2026-03-12 01:09:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
load=149689]: Uploading piece...
Mar 12 00:35:45.870157663 (133h35m43.466928696s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.870748221 (133h35m43.467519249s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871520585 (133h35m43.468291614s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.871619897 (133h35m43.468390931s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872089773 (133h35m43.468860803s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872223558 (133h35m43.468994582s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872522029 (133h35m43.469293052s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.872981308 (133h35m43.469752334s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873519843 (133h35m43.470290872s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.873936881 (133h35m43.470707904s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874299912 (133h35m43.471070938s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.874751304 (133h35m43.47152233s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876388472 (133h35m43.473159495s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.876896905 (133h35m43.473667929s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.878804068 (133h35m43.47557509s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:45.879710862 (133h35m43.47648188s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Uploading piece...
Mar 12 00:35:49.930529953 (133h35m47.527300969s) [upload=149689]: Next returned nil segment
Mar 12 00:35:49.930545799 (133h35m47.527316822s) [upload=149689]: Waiting for error group managing segments...
Mar 12 00:35:50.898312063 (133h35m48.495083094s) [noise=true , address=212.102.52.214:4346 , piece_id=UJF4FPZMBR5GQ6MHQTF5GTCTNNVAWJM2L2SXW7GGNJAQNAIUHD2Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.905765918 (133h35m48.502536941s) [noise=true , address=212.102.52.216:4402 , piece_id=KOQBHTM3UMBRFZQMI65ULLFEFYYJOHFSVB55HZTHCTCBLEUOZ65A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:50.915640983 (133h35m48.512412012s) [noise=true , address=212.102.52.214:4349 , piece_id=W6SRV76BQ6NE2IEXCEF732RUEQKKAEMNJEDURJFVJLBKGOLMJM5A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:51.625369720 (133h35m49.222140746s) [noise=true , address=212.102.52.213:4321 , piece_id=2F27ZXLXUKYBR7APN2HMCCSPMX7Q2FEOKZRY6QCXZQ3BGMZTETTQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.435650710 (133h35m50.032421722s) [noise=true , address=212.102.52.215:4372 , piece_id=4A6SLM4LTCNKAFIL4S6SQGI7MBAB2K6IGLSTRXQJ3F3CMUAVUWZA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.444469444 (133h35m50.04124047s) [noise=true , address=212.102.52.217:4431 , piece_id=DSK2GALEE2OTVNT4OYUUHNSDN56PZAOFTTNVU7NIV6BF7K7XPT4A, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453738377 (133h35m50.050509389s) [noise=true , address=212.102.52.215:4371 , piece_id=JN6SOBRAQRPSP3KUHHBQZWPMOZ4KPXCPAAW4ZOB2BPL6WRBOSHEA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.453807128 (133h35m50.050578143s) [noise=true , address=212.102.52.216:4400 , piece_id=YAUISSLBOMFIABHJTNZPSEPORYWCMOM55VQQDLSPL6YWCGZJIP4Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:52.462812772 (133h35m50.059583795s) [noise=true , address=212.102.52.216:4415 , piece_id=ON56HAWJ3XSQCCJGL7NRRTJVZRQZZN2ZO5UK6EU567VPRGUV62VA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.747909234 (133h35m51.344680258s) [noise=true , address=212.102.52.217:4434 , piece_id=3VCTW2BQXOTHQORAEYN5TWL5GA77EOL2NMHZ43MY5GIPKDSXMTBA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.748044679 (133h35m51.344815706s) [noise=true , address=212.102.52.217:4430 , piece_id=4ECEHEWBLCTYEA5MM2JNBCKOGEGSAGVZ4EFGZJFUAD36AWPXLKYQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.753942765 (133h35m51.350713789s) [noise=true , address=212.102.52.215:4370 , piece_id=BQHUQOLZBDNH3ZRMJFFQNQJMZR3Q4EPQIEA4CNHSQN6RHPR6ZQZQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:53.767093543 (133h35m51.363864567s) [noise=true , address=212.102.52.214:4351 , piece_id=OPLAHJ6XB5VIKSIEE2Q7DUM2JKDO4CG5ZQ6FHZWJSNMTB3GDHKGQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.534761485 (133h35m52.13153252s) [noise=true , address=212.102.52.217:4436 , piece_id=XWWFBBMCBZXJ2USWPHO5GRRDTVZITDQPJTJIIE3NAI5OX27L4UUA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.546709062 (133h35m52.143480086s) [noise=true , address=212.102.52.214:4353 , piece_id=L44Z7EQGYJ5KW6TMIKL6JDFDHZZL6T5A3OQTSXP4OOORPI7CZHOQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547405013 (133h35m52.144176036s) [noise=true , address=212.102.52.213:4344 , piece_id=YPXLDXNHVRE6JXF3V6HAXVN2LM7KGAA45BN4CS7QCD5ZNKXVOOUQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547419063 (133h35m52.144190093s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 16 pieces
Mar 12 00:35:54.547677884 (133h35m52.144448918s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9pEghsBzskA6HC7HMDNF7XhJEQFvxBRx95cT7E2RuMxegnNz): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547687719 (133h35m52.144458777s) [noise=true , address=212.102.52.217:4437 , piece_id=L75D5KJNU4ADWD4LERHD5CCHMCSLT3AVDQZPS26OZOYKE32E7PVQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547704947 (133h35m52.144475971s) [noise=true , address=212.102.52.213:4325 , piece_id=B4H67AGO7PAUGH5KUG75GTOK62G7CAYM2B62YAYNMUBGLD72V67Q, seg_pos={0 0}, upload=149689]: Done uploading piece. err: <nil>
Mar 12 00:35:54.547722444 (133h35m52.144493466s) [seg_pos={0 0}, upload=149689]: Satisfied cohort requirements after 17 pieces
Mar 12 00:35:54.547834736 (133h35m52.144605769s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cwAu1rBbxVBvbgfqzFyWSjF8t6Si8Z32BLiHuVzvkAGbLb7TR): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547842605 (133h35m52.144613631s) [noise=true , address=212.102.52.214:4362 , piece_id=DLV7NZEMY3DIKKMUFHI7QTAVO322TLGS4ULUPXFJGVZOPYP2JQHA, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547964853 (133h35m52.14473588s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fAGLyjawu1GJ6crrZAEJxVuVwEUHYnUNkvcden4QkK53cVwPy): protocol: expected piece hash; context canceled; EOF
Mar 12 00:35:54.547971598 (133h35m52.144742624s) [noise=true , address=212.102.52.216:4399 , piece_id=DNIGRHXLGRBQ6FR77SFE5ADJ2GBUJI3CHCPQQXBXTCB3YEXZQLPQ, seg_pos={0 0}, upload=149689]: Long tail cancellation due to optimal threshold
Mar 12 00:35:54.547992617 (133h35m52.14476363s) [seg_pos={0 0}, upload=149689]: Done waiting for segment. successful: 17 optimal: 16 errs: <nil>
Mar 12 00:35:54.548083620 (133h35m52.144854647s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 12 00:35:54.548090735 (133h35m52.144861761s) [upload=149689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 12 00:35:54.682468585 (133h35m52.279239608s) [upload=149689]: Done uploading segments...
Mar 12 00:35:54.682489239 (133h35m52.279260263s) [upload=149689]: Upload finished. err: <nil>
Duration: 1.976841812s
core
uplink
uk-select-1
uk1-placement-17
|