Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 22 out of 60 is failed (threshold=3) [.....................................X.XXXXXXXXXXXXXXXXXXXXX] duration: 30.00056208s FAIL (2025-08-07 21:30:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 48825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000581639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.001551033s FAIL (2025-08-07 21:30:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001568236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.002060852s FAIL (2025-08-07 21:30:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002089337s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.001390021s FAIL (2025-08-07 21:30:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001432927s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 23 out of 60 is failed (threshold=3) [...................................X..XXXXXXXXXXXXXXXXXXXXXX] duration: 30.000491205s FAIL (2025-08-07 21:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000510391s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 1m0.000780905s FAIL (2025-08-07 21:30:10 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 70:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.620380327 (11h39m30.135482901s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.621689932 (11h39m30.136792523s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.622549540 (11h39m30.137652109s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.623266213 (11h39m30.138368781s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.623648222 (11h39m30.138750801s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.630084181 (11h39m30.145186749s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.636336554 (11h39m30.151439122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.638431947 (11h39m30.15353452s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.640676036 (11h39m30.155778606s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.642585356 (11h39m30.157687924s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.643067815 (11h39m30.158170386s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.703545936 (11h39m30.218648502s) [upload=12100]: Next returned nil segment Aug 7 21:18:55.703561907 (11h39m30.218664489s) [upload=12100]: Waiting for error group managing segments... Aug 7 21:18:55.722257588 (11h39m30.237360157s) [noise=true , address=45.91.116.7:4020 , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.724853022 (11h39m30.239955588s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.725037274 (11h39m30.240139844s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730399702 (11h39m30.245502269s) [noise=true , address=104.36.82.2:4747 , piece_id=ZRKZEN5KXEHMD3OBPISFWZMLA7H2YNKUB5J57IVQ5AVHA5IJGXGQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730432338 (11h39m30.2455349s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.739954757 (11h39m30.255057333s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.742133144 (11h39m30.257235706s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.745676444 (11h39m30.260779013s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.747051082 (11h39m30.262153652s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces. Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments... Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments... Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments... Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments... Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded Duration: 1m0.000805192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.000696147s FAIL (2025-08-07 21:30:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000716414s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [......................................X.XXXXXXXXXXXXXXXXXXXX] duration: 30.001453725s FAIL (2025-08-07 21:30:03 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001479976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.00177465s FAIL (2025-08-07 21:30:02 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001798251s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXX........XX.] duration: 239.909396ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:29:55 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 239.926414ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......................................X..XXXXXXXXXXXXXXXXXX] duration: 30.000796415s FAIL (2025-08-07 21:29:54 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== load=12153]: Next segment err: context deadline exceeded Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments... Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments... Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments... Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments... Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments... Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments... Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Duration: 30.000813188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.0007171s FAIL (2025-08-07 21:29:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000746288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.000803041s FAIL (2025-08-07 21:29:45 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000820373s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 20 out of 60 is failed (threshold=3) [.......................................X.XXXXXXXXXXXXXXXXXXX] duration: 30.001253924s FAIL (2025-08-07 21:29:30 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 18654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:00.144941224 (11h49m34.660043804s) [upload=12267]: Starting upload Aug 7 21:29:00.145039156 (11h49m34.660141718s) [upload=12267]: Done starting upload Aug 7 21:29:00.145159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001273534s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [....................................X..XXXXXXXXXXXXXXXXXXXXX] duration: 30.001460775s FAIL (2025-08-07 21:29:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002075332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........................................X..XXXXXXXXXXXXXXXXX] duration: 30.000491715s FAIL (2025-08-07 21:29:24 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== load=12142]: Next segment err: context deadline exceeded Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments... Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments... Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments... Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments... Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments... Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments... Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments... Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments... Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Duration: 30.000513493s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m0.000585071s FAIL (2025-08-07 21:29:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000617822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.001481103s FAIL (2025-08-07 21:29:15 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001493313s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 1m0.000730764s FAIL (2025-08-07 21:28:57 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 18:55.615058039 (11h39m30.130160608s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.615957943 (11h39m30.13106051s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.617657344 (11h39m30.13275991s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.618525740 (11h39m30.133628306s) [noise=true , address=45.91.116.7:4020 , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.619948388 (11h39m30.135050957s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.620380327 (11h39m30.135482901s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.621689932 (11h39m30.136792523s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.622549540 (11h39m30.137652109s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.623266213 (11h39m30.138368781s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.623648222 (11h39m30.138750801s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.630084181 (11h39m30.145186749s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.636336554 (11h39m30.151439122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.638431947 (11h39m30.15353452s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.640676036 (11h39m30.155778606s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.642585356 (11h39m30.157687924s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.643067815 (11h39m30.158170386s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.703545936 (11h39m30.218648502s) [upload=12100]: Next returned nil segment Aug 7 21:18:55.703561907 (11h39m30.218664489s) [upload=12100]: Waiting for error group managing segments... Aug 7 21:18:55.722257588 (11h39m30.237360157s) [noise=true , address=45.91.116.7:4020 , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.724853022 (11h39m30.239955588s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.725037274 (11h39m30.240139844s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730399702 (11h39m30.245502269s) [noise=true , address=104.36.82.2:4747 , piece_id=ZRKZEN5KXEHMD3OBPISFWZMLA7H2YNKUB5J57IVQ5AVHA5IJGXGQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730432338 (11h39m30.2455349s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.739954757 (11h39m30.255057333s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.742133144 (11h39m30.257235706s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.745676444 (11h39m30.260779013s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.747051082 (11h39m30.262153652s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces. Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments... Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments... Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded Duration: 1m0.000746317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.000400141s FAIL (2025-08-07 21:28:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000417843s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.00155287s FAIL (2025-08-07 21:28:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001575361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.....................................X..XXXXXXXXXXXXXXXXXXXX] duration: 30.000735593s FAIL (2025-08-07 21:28:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000760525s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.000602027s FAIL (2025-08-07 21:28:49 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000645459s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........................................X..XXXXXXXXXXXXXXXX] duration: 30.001330441s FAIL (2025-08-07 21:28:45 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ) [upload=12126]: Next segment err: context deadline exceeded Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments... Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments... Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments... Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments... Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments... Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments... Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments... Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments... Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments... Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments... Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Duration: 30.0013558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 30.000473761s FAIL (2025-08-07 21:28:44 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00049685s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 44.805838ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:28:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 44.828383ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [........................................X.XXXXXXXXXXXXXXXXXX] duration: 30.001308984s FAIL (2025-08-07 21:28:29 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:59.764993056 (11h48m34.280095625s) [upload=12250]: Starting upload Aug 7 21:27:59.765094736 (11h48m34.280197302s) [upload=12250]: Done starting upload Aug 7 21:27:59.765177450 (11h48m34.280280017s) [upload=12250]: Uploading segments... Aug 7 21:27:59.765224668 (11h48m34.280327253s) [upload=12250]: Got next segment. Inline: true Aug 7 21:27:59.765229860 (11h48m34.280332436s) [upload=12250]: Waiting for error group managing segments... Aug 7 21:27:59.765296818 (11h48m34.280399397s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:59.765302009 (11h48m34.280404586s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:59.765305267 (11h48m34.280407844s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:29.730125599 (11h49m4.245228185s) [upload=12250]: Done uploading segments... Aug 7 21:28:29.730155654 (11h49m4.245258236s) [upload=12250]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001352239s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.000714641s FAIL (2025-08-07 21:28:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000735655s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.002030169s FAIL (2025-08-07 21:28:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002047345s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 45.055557ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:28:17 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 45.084245ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.001319991s FAIL (2025-08-07 21:28:12 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001427066s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.001159283s FAIL (2025-08-07 21:28:09 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001173518s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 20 out of 60 is failed (threshold=3) [......................................X..XXXXXXXXXXXXXXXXXXX] duration: 30.00165061s FAIL (2025-08-07 21:28:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00170264s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 9m51.975134484s FAIL (2025-08-07 21:27:57 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== next segment. Inline: false Aug 7 21:18:05.960789068 (11h38m40.475891653s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:05.960795140 (11h38m40.475897716s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:55.613300690 (11h39m30.128403266s) [seg_pos={0 0}, upload=12100]: Begin upload segment... Aug 7 21:18:55.613383589 (11h39m30.128486161s) [seg_pos={0 0}, upload=12100]: Done begin upload segment. Aug 7 21:18:55.614114361 (11h39m30.129216939s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.614197037 (11h39m30.129299623s) [noise=true , address=104.36.82.2:4747 , piece_id=ZRKZEN5KXEHMD3OBPISFWZMLA7H2YNKUB5J57IVQ5AVHA5IJGXGQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.615058039 (11h39m30.130160608s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.615957943 (11h39m30.13106051s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.617657344 (11h39m30.13275991s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.618525740 (11h39m30.133628306s) [noise=true , address=45.91.116.7:4020 , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.619948388 (11h39m30.135050957s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.620380327 (11h39m30.135482901s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.621689932 (11h39m30.136792523s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.622549540 (11h39m30.137652109s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.623266213 (11h39m30.138368781s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.623648222 (11h39m30.138750801s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.630084181 (11h39m30.145186749s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.636336554 (11h39m30.151439122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.638431947 (11h39m30.15353452s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.640676036 (11h39m30.155778606s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.642585356 (11h39m30.157687924s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.643067815 (11h39m30.158170386s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Uploading piece... Aug 7 21:18:55.703545936 (11h39m30.218648502s) [upload=12100]: Next returned nil segment Aug 7 21:18:55.703561907 (11h39m30.218664489s) [upload=12100]: Waiting for error group managing segments... Aug 7 21:18:55.722257588 (11h39m30.237360157s) [noise=true , address=45.91.116.7:4020 , piece_id=RLJSC76ICPPSAK4ZFZPRD2GQP54VESM4RSXMX4EOZTU5LNYDWTZA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.724853022 (11h39m30.239955588s) [noise=true , address=104.36.82.62:4414 , piece_id=R2IV3BPQQX7JNXHJYPLOSMWVAGNX7P47YGRK7ID7D3TI3BQPEFDQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.725037274 (11h39m30.240139844s) [noise=true , address=45.91.116.9:4273 , piece_id=ZKYGGUFRFOLQZ6SWGVNPLDRBSZJDG7YK3L7Y4C7Y5ECK5Q2IMNQQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730399702 (11h39m30.245502269s) [noise=true , address=104.36.82.2:4747 , piece_id=ZRKZEN5KXEHMD3OBPISFWZMLA7H2YNKUB5J57IVQ5AVHA5IJGXGQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.730432338 (11h39m30.2455349s) [noise=true , address=104.36.82.60:4359 , piece_id=ZNFCBAV5YUEDTPNW75CPMYH22TE2XQUGEBNCAGUCSKGJCNBG6FJA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.739954757 (11h39m30.255057333s) [noise=true , address=45.91.116.8:4064 , piece_id=T56UKN2YJFJ5HJQZAADSAAL4WCGFRTXCHUIBWUASAWXJCNKBWFWQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.742133144 (11h39m30.257235706s) [noise=true , address=45.91.116.11:4139 , piece_id=AKRKBNB72JBXDVOJSKULPRENHWHYMB3RH6O4PIBEZM2JKCBI5D5A, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.745676444 (11h39m30.260779013s) [noise=true , address=104.36.82.58:4272 , piece_id=EVWQ54CZE3Z7ZNFK3DHXNRLZCSRXZCBYHMUAIZJXS2ESYHB7IV3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.747051082 (11h39m30.262153652s) [noise=true , address=104.36.82.15:4287 , piece_id=4GNHDSQUKPUMBIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil> Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces. Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Duration: 9m51.97515403s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 34.416048ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:27:56 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 34.440034ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 18 out of 60 is failed (threshold=3) [.........................................X.XXXXXXXXXXXXXXXXX] duration: 30.00121355s FAIL (2025-08-07 21:27:54 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:24.082403775 (11h47m58.597506352s) [upload=12238]: Starting upload Aug 7 21:27:24.082490140 (11h47m58.597592716s) [upload=12238]: Done starting upload Aug 7 21:27:24.082558940 (11h47m58.59766151s) [upload=12238]: Uploading segments... Aug 7 21:27:24.082629361 (11h47m58.597731945s) [upload=12238]: Got next segment. Inline: true Aug 7 21:27:24.082635193 (11h47m58.59773777s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:27:24.082691002 (11h47m58.597793571s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:24.082696855 (11h47m58.597799431s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:24.082700406 (11h47m58.597802972s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:54.030733770 (11h48m28.545836347s) [upload=12238]: Done uploading segments... Aug 7 21:27:54.030757764 (11h48m28.545860341s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001237201s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXX........X] duration: 2m0.001000941s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:27:54 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.001024036s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.000999695s FAIL (2025-08-07 21:27:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001013936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........................................X..XXXXXXXXXXXXXXX] duration: 30.001323152s FAIL (2025-08-07 21:27:48 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ) [upload=12116]: Next segment err: context deadline exceeded Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments... Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments... Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments... Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments... Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments... Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments... Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments... Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments... Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments... Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments... Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments... Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments... Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Aug 7 21:27:18.369066326 (11h47m52.884168905s) [upload=12236]: Starting upload Aug 7 21:27:18.369135373 (11h47m52.884237939s) [upload=12236]: Done starting upload Aug 7 21:27:18.369525932 (11h47m52.884628509s) [upload=12236]: Uploading segments... Aug 7 21:27:18.369615706 (11h47m52.884718294s) [upload=12236]: Got next segment. Inline: false Aug 7 21:27:18.369678027 (11h47m52.88478061s) [upload=12236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.369684686 (11h47m52.884787271s) [upload=12236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.337977385 (11h48m22.853079968s) [upload=12236]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Duration: 30.001353396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.001110682s FAIL (2025-08-07 21:27:39 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001134s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.......................................X..XXXXXXXXXXXXXXXXXX] duration: 30.000681329s FAIL (2025-08-07 21:27:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000700764s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.000825585s FAIL (2025-08-07 21:27:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000846259s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.00059203s FAIL (2025-08-07 21:27:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000627501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 17 out of 60 is failed (threshold=3) [..........................................X.XXXXXXXXXXXXXXXX] duration: 30.001062102s FAIL (2025-08-07 21:27:09 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:39.360574327 (11h47m13.875676912s) [upload=12228]: Starting upload Aug 7 21:26:39.360670988 (11h47m13.875773563s) [upload=12228]: Done starting upload Aug 7 21:26:39.360747494 (11h47m13.875850071s) [upload=12228]: Uploading segments... Aug 7 21:26:39.360815021 (11h47m13.875917612s) [upload=12228]: Got next segment. Inline: true Aug 7 21:26:39.360820926 (11h47m13.875923502s) [upload=12228]: Waiting for error group managing segments... Aug 7 21:26:39.360871797 (11h47m13.875974374s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:39.360877512 (11h47m13.875980091s) [upload=12228]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:39.360882268 (11h47m13.875984845s) [upload=12228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.331560508 (11h47m43.846663074s) [upload=12228]: Done uploading segments... Aug 7 21:27:09.331579927 (11h47m43.846682494s) [upload=12228]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001076953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........................................X..XXXXXXXXXXXXXX] duration: 30.001103185s FAIL (2025-08-07 21:27:02 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== lush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments... Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments... Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments... Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments... Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments... Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments... Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments... Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments... Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments... Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments... Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments... Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments... Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments... Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments... Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments... Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments... Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments... Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments... Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments... Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments... Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments... Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.978197373 (11h47m7.49329995s) [upload=12224]: Starting upload Aug 7 21:26:32.978293594 (11h47m7.493396174s) [upload=12224]: Done starting upload Aug 7 21:26:32.978433198 (11h47m7.493535776s) [upload=12224]: Uploading segments... Aug 7 21:26:32.978843092 (11h47m7.493945723s) [upload=12224]: Got next segment. Inline: false Aug 7 21:26:32.978911769 (11h47m7.494014345s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.978917147 (11h47m7.494019724s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:02.946202472 (11h47m37.461305058s) [upload=12224]: Next segment err: context deadline exceeded Aug 7 21:27:02.946251711 (11h47m37.461354288s) [upload=12224]: Done uploading segments... Aug 7 21:27:02.946270703 (11h47m37.461373272s) [upload=12224]: Upload finished. err: context deadline exceeded Duration:* 30.001130485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.001333686s FAIL (2025-08-07 21:26:59 UTC) operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001354209s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 125.844886ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:26:55 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 125.87459ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.000746108s FAIL (2025-08-07 21:26:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000766605s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.000755331s FAIL (2025-08-07 21:26:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 35 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:50.700806744 (11h46m25.215909321s) [upload=12213]: Starting upload Aug 7 21:25:50.700881110 (11h46m25.215983679s) [upload=12213]: Done starting upload Aug 7 21:25:50.700953663 (11h46m25.216056233s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000783335s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 18 out of 60 is failed (threshold=3) [........................................X..XXXXXXXXXXXXXXXXX] duration: 30.002309305s FAIL (2025-08-07 21:26:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00233406s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)