Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.055290454s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:16:14 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.055320075s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................X.......................] duration:
897.186454ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:16:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 897.219769ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................X......................] duration:
916.346319ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:15:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 916.39941ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................................X.....................] duration:
794.743831ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:15:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 794.852011ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
1.967557464s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:15:06 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.967605808s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................................X....................] duration:
969.650831ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:14:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 969.675579ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.993550948s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:13:59 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.993599602s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................................X...................] duration:
867.209981ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:13:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 867.306516ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................X..................] duration:
924.00798ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:13:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 924.045504ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [X....X......................................................] duration:
1.929598158s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:13:24 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.929638824s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................X.................] duration:
1.188432695s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:12:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.188477332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration:
2.008068524s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:12:11 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.008099832s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................X................] duration:
1.466931354s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:11:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.466986067s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............................................X...............] duration:
767.101342ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:11:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 767.131681ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..X....X....................................................] duration:
2.02065513s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:10:57 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.020685528s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............................................X..............] duration:
1.15458894s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:10:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.154608914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................................X.............] duration:
876.176718ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:10:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 876.207294ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...X....X...................................................] duration:
2.00427803s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:09:50 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.004302945s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [....X....X..................................................] duration:
2.100756633s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:09:06 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.100794354s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............................................X............] duration:
782.994996ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:08:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 783.029844ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration:
2.08505682s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:08:45 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.085113678s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................................X...........] duration:
737.633637ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:08:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 737.657206ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [......X....X................................................] duration:
2.107534637s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:08:03 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.107569423s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................................X..........] duration:
2.414128968s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:07:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.414158157s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration:
3.197720915s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:07:43 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 3.197993786s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........] duration:
881.645334ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:07:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 881.676991ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [........X....X..............................................] duration:
2.116590986s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:07:22 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.116639216s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........] duration:
1.260658729s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:06:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.260687602s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration:
1.892714826s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:06:10 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.892750147s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......] duration:
1.033669835s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:06:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.033707125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..........X....X............................................] duration:
2.034461872s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:05:55 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.034526027s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......] duration:
1.275011184s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:05:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.275051286s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration:
1.963173858s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:04:45 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.963209404s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....] duration:
1.388653528s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:04:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.38871886s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................................................X....] duration:
3.352656135s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:04:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.352699675s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [............X....X..........................................] duration:
2.126745091s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:04:02 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.126771877s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
2.232305812s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:03:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.232335024s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration:
2.060566193s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:03:10 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.060597188s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
838.907567ms trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:02:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 838.93679ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..............X....X........................................] duration:
1.915911108s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:02:20 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.915939158s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
1.007164612s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:01:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.007259356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.001779186s trace:
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:01:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001797121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration:
1.982199226s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:01:30 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.98223469s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [................X....X......................................] duration:
7.371084044s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:00:41 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 7.371119928s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration:
1.97781968s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:00:00 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.977849533s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..................X....X....................................] duration:
2.065653593s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-25 23:59:31 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.065683704s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...................X....X...................................] duration:
4.099710778s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-25 23:59:15 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 4.099749963s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [....................X....X..................................] duration:
1.974996671s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-25 23:58:11 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 1.975025893s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.....................X....X.................................] duration:
2.106856766s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-25 23:57:46 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 2.106876665s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [......................X....X................................] duration:
5.816974862s trace:
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-25 23:56:35 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337 , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421 , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375 , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385 , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368 , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362 , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396 , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389 , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433 , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374 , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400 , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326 , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347 , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395 , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410 , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325 , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346 , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427 , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438 , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324 , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>
Duration: 5.817006613s
core
uplink
uk-select-1
uk1-placement-17
|