Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

25 Active (Non-Silenced) Services (0 down, 1 warning)
Report: 2026-05-26 00:14:44 UTC
Toggle:
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60):
antiblip: 
1 out of 60 is failed (threshold=6) [.......................................X....................]
duration: 
969.650831ms
trace: 
https://jaeger.dev.storj.io/trace/72dae83432a6990d
FAIL (2026-05-26 00:14:44 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
	storj.io/orbiter/checkers.(*Gateway).Check:413
	storj.io/orbiter/metrics.(*CheckerMetrics).Check:82
	storj.io/orbiter/checkers.(*AntiBlip).checkDelegate:61
	storj.io/orbiter/checkers.(*AntiBlip).Check:76
	storj.io/orbiter/service.(*Jittered).Run.func1:75
	storj.io/orbiter/service.(*Jittered).Run:76
	storj.io/orbiter/registry.(*Registry).Run.(*Registry).Run.func1.func2:76
	golang.org/x/sync/errgroup.(*Group).Go.func1:93
Trigger core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60):
antiblip: 
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration: 
1.993550948s
trace: 
https://jaeger.dev.storj.io/trace/759482bf7b00ff11
FAIL (2026-05-26 00:13:59 UTC) 
orbiter warning: 
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
273794845s) [noise=true , address=212.102.52.215:4375  , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102049571 (604h42m32.274168006s) [noise=true , address=212.102.52.217:4433  , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102460462 (604h42m32.274578882s) [noise=true , address=212.102.52.216:4410  , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.102872035 (604h42m32.274990454s) [noise=true , address=212.102.52.215:4389  , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103278596 (604h42m32.275397018s) [noise=true , address=212.102.52.214:4346  , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.103804989 (604h42m32.275923414s) [noise=true , address=212.102.52.216:4400  , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104247382 (604h42m32.276365803s) [noise=true , address=212.102.52.217:4438  , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.104888697 (604h42m32.277007127s) [noise=true , address=212.102.52.213:4324  , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.105707663 (604h42m32.277826093s) [noise=true , address=212.102.52.214:4362  , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.106495770 (604h42m32.278614199s) [noise=true , address=212.102.52.216:4395  , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.107393880 (604h42m32.279512312s) [noise=true , address=212.102.52.214:4368  , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.108359828 (604h42m32.280478258s) [noise=true , address=212.102.52.217:4421  , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.110405093 (604h42m32.282523524s) [noise=true , address=212.102.52.213:4325  , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.112553307 (604h42m32.284671735s) [noise=true , address=212.102.52.213:4337  , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.114921518 (604h42m32.287039952s) [noise=true , address=212.102.52.217:4427  , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Uploading piece...
May 25 23:31:20.606427910 (604h42m32.778546337s) [upload=677090]: Next returned nil segment
May 25 23:31:20.606509190 (604h42m32.778627615s) [upload=677090]: Waiting for error group managing segments...
May 25 23:31:20.745095998 (604h42m32.917214426s) [noise=true , address=212.102.52.213:4337  , piece_id=UMENVWGZAOTC367LV6JYPEYA2PB6GWWOG62ZYJUNJHGUFPMUF5UA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.745839039 (604h42m32.917957467s) [noise=true , address=212.102.52.217:4421  , piece_id=IKYNA6P2CMV3NGN5YOBIKFEK5XG4VIA7KOFDYUDSUYNONUUO4RQQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.746514239 (604h42m32.918632674s) [noise=true , address=212.102.52.215:4375  , piece_id=LZX2COO3QPMO3JBDW2GDPXJ6HQYEZMYAW7L3PRO26O4RSOU6WF6A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747200040 (604h42m32.919318463s) [noise=true , address=212.102.52.215:4385  , piece_id=BYAOM6QCVRNMM2LUBKJBC624PAT4FJNYZT33OAJ64XUX2RIWUCQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.747877259 (604h42m32.919995693s) [noise=true , address=212.102.52.214:4368  , piece_id=DZGK5FE3JFKNX7627B6YKR3YCBSI44GDJZYLEDUKRE7B54KAZTKQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.749745071 (604h42m32.921863504s) [noise=true , address=212.102.52.214:4362  , piece_id=WS2PQ7CUBLUHX5UMPWHVF7DGU7RMIWNBNQMQ66QDWIKYIOJK6WYA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750305240 (604h42m32.922423662s) [noise=true , address=212.102.52.216:4396  , piece_id=EFOMMQ353UU3A23MUNAT45DFLGAWRSEQFL5MFHXLU4SOOIZ6WGIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.750808014 (604h42m32.922926426s) [noise=true , address=212.102.52.215:4389  , piece_id=4IEEL3PUN5VNX3RNX6GTPC7J6KQCP6XCT46CWVTMIORCFQ4YA37Q, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752630121 (604h42m32.924748548s) [noise=true , address=212.102.52.217:4433  , piece_id=DMXSYTKH3RGDRFIIXBKKK5U5K5LSV4ZAA4H6V72DXMSGCYJQ54GQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.752969937 (604h42m32.925088365s) [noise=true , address=212.102.52.215:4374  , piece_id=NZF7MCAN57HFGOYIGAWRJTCDXCQSZ6JTTFUSZQUJSS72YPMV2YIA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753257861 (604h42m32.925376289s) [noise=true , address=212.102.52.216:4400  , piece_id=VOETMF7IEC4TJ7Y3SJMYZ7LQUWPUOXRSCJ7JF7X36F7LNRWQZKCQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.753934765 (604h42m32.926053192s) [noise=true , address=212.102.52.213:4326  , piece_id=YIZFEYZVSUXZ4PBFG6KHSA6ALEANBKOYZHGTFDXNI3F4TDEWCTXA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754070405 (604h42m32.926188836s) [noise=true , address=212.102.52.214:4347  , piece_id=T5U5YXI6VEWHKOQPAPKD7LFVHWESRTCKQPIQAPU5WXS5IAI334ZA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.754630870 (604h42m32.926749295s) [noise=true , address=212.102.52.216:4395  , piece_id=IQJJOJQICPELKKV6ZIZKSV34JTYQXGXTBWV36H7URMVWEJ257C2A, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755096499 (604h42m32.927214932s) [noise=true , address=212.102.52.216:4410  , piece_id=IZPCVKUERZOAIHONQBTMDUQVWCUUJRUAFTV3UHNLK43HNG7JUKQA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755205664 (604h42m32.927324091s) [noise=true , address=212.102.52.213:4325  , piece_id=BTZVAK3WPPRI5TMGOYELHCMXX4C3OYFNGWK56LI4AXALG6XVESHQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: <nil>
May 25 23:31:20.755220281 (604h42m32.927338707s) [seg_pos={0 0}, upload=677090]: Satisfied cohort requirements after 16 pieces
May 25 23:31:20.755486971 (604h42m32.9276054s) [noise=true , address=212.102.52.214:4346  , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BA6ZmPnVWhQ86R27YBcNR2f7qDAbjPBadMtYxvzBpDcunYt8F): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755496821 (604h42m32.92761525s) [noise=true , address=212.102.52.214:4346  , piece_id=6NLD46HAV4HQIUJFPMNEBWDKKESPI36IMUMIVO7EHB634SFLTOHA, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755774216 (604h42m32.927892656s) [noise=true , address=212.102.52.217:4427  , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mHkZ2yrJaVP3EbaU7YcSCzhcLHCJ58mcYYJpCamzs6oCBErfp): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755784384 (604h42m32.927902816s) [noise=true , address=212.102.52.217:4427  , piece_id=NOCLIOJQI7TKU3SJRK4J3D6AT4HKZ6CK6MHENAXE7TI2STSH5XXQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.755967780 (604h42m32.92808621s) [noise=true , address=212.102.52.217:4438  , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ugtnw8cWHpnngeVtPDo8UV3nXBtP2Gx3hDkEeepDJrirnz8As): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.755975984 (604h42m32.928094414s) [noise=true , address=212.102.52.217:4438  , piece_id=R5LBQJQZT2MAKAGGBT5C2RLB2M4MN7BWMPZQCUMSBYZJRQEUPNLQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756136257 (604h42m32.928254686s) [noise=true , address=212.102.52.213:4324  , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUFu2hhfUxzYsmwuszgkGd2ApBBpkttz1STJRyp9eB73R8D6B): protocol: expected piece hash; context canceled; EOF
May 25 23:31:20.756144610 (604h42m32.92826304s) [noise=true , address=212.102.52.213:4324  , piece_id=4LEUOFALSCOY6MMPZQYMEMRNFDW7V4VBR2IUTJA4MKUORN6H5KYQ, seg_pos={0 0}, upload=677090]: Long tail cancellation due to optimal threshold
May 25 23:31:20.756169252 (604h42m32.928287678s) [seg_pos={0 0}, upload=677090]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 23:31:20.756297694 (604h42m32.928416123s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 23:31:20.756304829 (604h42m32.928423255s) [upload=677090]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 23:31:53.390837206 (604h43m5.562955636s) [upload=677090]: Done uploading segments...
May 25 23:31:53.390872455 (604h43m5.562990863s) [upload=677090]: Upload finished. err: <nil>

	storj.io/orbiter/checkers.(*Uplink).formatErrWithLogs:185
	storj.io/orbiter/checkers.(*Uplink).Check:142
	storj.io/orbiter/service.(*Silencer).Check:25
	storj.io/orbiter/metrics.(*CheckerMetrics).Check:82
	storj.io/orbiter/checkers.(*AntiBlip).checkDelegate:61
	storj.io/orbiter/checkers.(*AntiBlip).Check:76
	storj.io/orbiter/service.(*Jittered).Run.func1:75
	storj.io/orbiter/service.(*Jittered).Run:76
	storj.io/orbiter/registry.(*Registry).Run.(*Registry).Run.func1.func2:76
	golang.org/x/sync/errgroup.(*Group).Go.func1:93
Trigger core uplink uk-select-1 uk1-placement-17 SILENCED