Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
806.918596ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:28:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 806.946327ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration:
2.068430071s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:28:54 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.068460989s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
847.333497ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:28:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 847.367145ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
705.579128ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:28:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 705.604624ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
835.086512ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:28:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 835.125191ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [........................X....X..............................] duration:
2.369748588s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:27:44 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.369782319s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
809.069543ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:26:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 809.160489ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration:
1.99475162s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:26:47 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 1.994784059s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
849.393438ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:26:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 849.540153ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................X.........................................] duration:
881.629728ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:25:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 881.666447ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..........................X....X............................] duration:
2.590125935s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:25:47 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.590160996s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration:
1.983412992s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:25:21 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 1.98364507s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [............................X....X..........................] duration:
4.384936613s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:24:55 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 4.384971828s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................X........................................] duration:
847.054569ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:24:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 847.118815ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration:
2.008454788s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:24:37 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.00848634s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..............................X....X........................] duration:
1.899112659s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:24:17 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 1.899164814s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................X.......................................] duration:
826.60359ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:23:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 826.670651ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration:
2.073440535s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:23:06 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.073556901s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................X......................................] duration:
830.879931ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:22:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 830.97506ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [................................X....X......................] duration:
2.424139779s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:22:49 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.424242894s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration:
2.565392003s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:22:16 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.565420773s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
804.885229ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:21:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 804.915694ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..................................X....X....................] duration:
2.099979441s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:21:36 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.100013372s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................X....................................] duration:
874.131703ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:21:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 874.159789ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...................................X....X...................] duration:
17.81628203s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:20:46 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 17.816343314s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................X...................................] duration:
834.009834ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:20:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 834.041726ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................X..................................] duration:
861.603832ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:20:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 861.636117ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................X.................................] duration:
778.760583ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:19:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 778.86636ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................X................................] duration:
886.216915ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:19:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 886.321131ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [....................................X....X..................] duration:
1.89727365s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:19:22 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 1.897299786s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration:
1.894978634s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:18:55 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 1.895013156s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............................X...............................] duration:
886.955379ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:18:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 886.984328ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............................X..............................] duration:
876.475558ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:18:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 876.493243ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [......................................X....X................] duration:
1.955967453s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:17:48 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 1.95600038s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................X.............................] duration:
738.994781ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:17:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 739.023884ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration:
2.080965697s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:16:53 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.080995876s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [........................................X....X..............] duration:
2.113187526s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:16:37 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.11321303s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............................X............................] duration:
833.788808ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:16:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 833.825375ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration:
2.177935537s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:15:57 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.177971153s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................X...........................] duration:
873.298839ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:15:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 873.330988ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X....X............] duration:
2.27652185s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:15:03 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.276569428s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................X..........................] duration:
863.642574ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:15:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 863.738209ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration:
1.90767866s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:14:16 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 1.907709332s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................X.........................] duration:
919.651917ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:13:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 919.71162ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [............................................X....X..........] duration:
2.061881732s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:13:48 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.06191136s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................X........................] duration:
841.528558ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:13:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 841.579524ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration:
2.20572754s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:12:35 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.205775603s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................X.......................] duration:
891.560235ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:12:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 891.630334ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X....X........] duration:
2.160071799s trace:
https://jaeger.dev.storj.io/trace/5e778fe7a13b60a0
FAIL (2026-05-25 11:12:07 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.096568326 (592h17m50.268686755s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097112562 (592h17m50.269230992s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.097632500 (592h17m50.269750931s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098148476 (592h17m50.270266905s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.098699871 (592h17m50.270818304s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099313777 (592h17m50.271432207s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.099840133 (592h17m50.271958563s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100162049 (592h17m50.272280479s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.100394430 (592h17m50.27251286s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101165405 (592h17m50.273283838s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101833189 (592h17m50.273951618s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.101859975 (592h17m50.273978416s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.102501215 (592h17m50.274619646s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103484544 (592h17m50.275602964s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.103611564 (592h17m50.275729993s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.104895143 (592h17m50.277013568s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.105575195 (592h17m50.277693619s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Uploading piece...
May 25 11:06:38.617322517 (592h17m50.789440943s) [upload=663150]: Next returned nil segment
May 25 11:06:38.617345221 (592h17m50.789463641s) [upload=663150]: Waiting for error group managing segments...
May 25 11:06:38.720245841 (592h17m50.892364266s) [noise=true , address=212.102.52.215:4374 , piece_id=VO2KM6U3AU3BGKVSEXBTI2775GZ4KSZYUZQ5HKQXLP3V7UEQ3ASQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.722028135 (592h17m50.894146559s) [noise=true , address=212.102.52.216:4397 , piece_id=TAHVCE4K2ALUN62UFFOYCAF7VE6OTUEVL7GNBIQNBR2SVT5ODY3Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.723587957 (592h17m50.895706388s) [noise=true , address=212.102.52.213:4339 , piece_id=JATZ2POQ4OP5U6ELUQNV6CAAMJE3633TOMQ3IXZJYTPMTJAGJE3A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.724810236 (592h17m50.89692866s) [noise=true , address=212.102.52.217:4438 , piece_id=6ICU5AYURZ2UM3CCC4ECAMMZX3W3JTBSMUIMGQNVMQVAO7BMKXRA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.725946605 (592h17m50.898065029s) [noise=true , address=212.102.52.217:4430 , piece_id=56MEVCZCL42LAB2H4P5DQ2E5DBNAQDG4BWR356ATM3YRSTHQ5ZHA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.728488369 (592h17m50.900606793s) [noise=true , address=212.102.52.217:4429 , piece_id=3O53KNSNQ2CJCKE57AK67YSWISWCGG7UNMPWACOSZDITSOLSBLWQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.733448559 (592h17m50.905567172s) [noise=true , address=212.102.52.214:4361 , piece_id=MZE3SCRQXKRMTPAHSR6A7RWTD4D477ZJRN3ZSI4IO4HHIJBX6XWA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.734401330 (592h17m50.906519754s) [noise=true , address=212.102.52.216:4406 , piece_id=F3XU7ZHGNOXMWKELLARRXHKQM3W6UGHR7YTWZJJJY7RVK47B7PCA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.735527451 (592h17m50.90764588s) [noise=true , address=212.102.52.215:4390 , piece_id=QX3AZASWXGRINW3R5PM4YSFIVB67JUU5M5PENMDMGTOF4JYHTB6Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.738774235 (592h17m50.910892658s) [noise=true , address=212.102.52.215:4378 , piece_id=WT2PJKH7UJLVGOD3MYKLZ3DFWGNUU6VKK7YURZDV4NCG4DUMO2MA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739476480 (592h17m50.911594889s) [noise=true , address=212.102.52.213:4332 , piece_id=KX7QRN6HEFVA45Q3M6PBGAR2PZPBHN2VMANQ4NGKNMWLFBFIIWOQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.739896594 (592h17m50.91201503s) [noise=true , address=212.102.52.213:4343 , piece_id=6TOF22V4QQZQQMXUNF4DZONHDLVSONSTYAQSQYJAYCVUGEF5OIXQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740257541 (592h17m50.912375965s) [noise=true , address=212.102.52.214:4363 , piece_id=5HDNLRWJB52UX52BWG5JUPBQUJIQXHYKEL2L6X2FP5FE3S3PA27Q, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.740598623 (592h17m50.912717043s) [noise=true , address=212.102.52.214:4364 , piece_id=M5OBC7WVS6RLFRVMCRZPEGWETTIGYQKVS7MOBFSFWNK5TNUFQLUA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741254827 (592h17m50.913373247s) [noise=true , address=212.102.52.217:4431 , piece_id=YZRB2E5L3AS2OKRD56U73BJV67INEK4JPANK3ZBENUZI7TCWNXDA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741939503 (592h17m50.914057909s) [noise=true , address=212.102.52.216:4411 , piece_id=LPJTMKTEEPF3Y4G6WSQ422DSQ53CQMKP5R4FBO6IQIZGPES3GDZQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.741954456 (592h17m50.914072881s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 16 pieces
May 25 11:06:38.742266991 (592h17m50.914385423s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16y12CqwtxrbMoBweh49rb4ZcgKgQqCNn3yApCJLQo3hJPBsoy): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.742278868 (592h17m50.914397275s) [noise=true , address=212.102.52.213:4326 , piece_id=H4SACSGFRP356NBXYLTMZEASDXJJBS6CWS44TXRIJRPRJM5BV7SQ, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.742882211 (592h17m50.915000634s) [noise=true , address=212.102.52.214:4358 , piece_id=HFPKCA3HYHFBXGLBPSZOGXZUYV4RWE5AP7KZ7FOMYPOUVXUMVJQA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.742899489 (592h17m50.915017913s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 17 pieces
May 25 11:06:38.743560409 (592h17m50.915678829s) [noise=true , address=212.102.52.215:4388 , piece_id=ZFKJUACOPRCDKD3MRTVNBL5N4WR3PUOM2FGRZQQKOSC6FYSQPJ4A, seg_pos={0 0}, upload=663150]: Done uploading piece. err: <nil>
May 25 11:06:38.743574497 (592h17m50.915692921s) [seg_pos={0 0}, upload=663150]: Satisfied cohort requirements after 18 pieces
May 25 11:06:38.747671019 (592h17m50.919789675s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 11:06:38.747692770 (592h17m50.919811199s) [noise=true , address=212.102.52.216:4416 , piece_id=NFP6MYIKDIO76Q25VRQHOKVS6KGW6QQI5E5WHITEGBSFM74BYGTA, seg_pos={0 0}, upload=663150]: Long tail cancellation due to optimal threshold
May 25 11:06:38.747725328 (592h17m50.919843735s) [seg_pos={0 0}, upload=663150]: Done waiting for segment. successful: 18 optimal: 16 errs: <nil>
May 25 11:06:38.747827383 (592h17m50.919946043s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 11:06:38.747874944 (592h17m50.919993376s) [upload=663150]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 11:06:45.559242654 (592h17m57.731361113s) [upload=663150]: Done uploading segments...
May 25 11:06:45.559267229 (592h17m57.731385653s) [upload=663150]: Upload finished. err: <nil>
Duration: 2.16010689s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................X......................] duration:
789.540951ms trace:
https://jaeger.dev.storj.io/trace/fb23a289728c702
FAIL (2026-05-25 11:11:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 790.930244ms
core
uplink
us-select-1
us1-placement-20
|