Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration:
1m0.000867987s
FAIL (2025-08-07 21:39:29 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== hRequestItem_ObjectCommit Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments... Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments... Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments... Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments... Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments... Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments... Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments... Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments... Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments... Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded Aug 7 21:32:19.033554133 (11h52m53.5486567s) [upload=12319]: Starting upload Aug 7 21:32:19.033620836 (11h52m53.548723412s) [upload=12319]: Done starting upload Aug 7 21:32:19.033708593 (11h52m53.548811183s) [upload=12319]: Uploading segments... Aug 7 21:32:19.033902998 (11h52m53.549005564s) [upload=12319]: Got next segment. Inline: false Aug 7 21:32:19.033935489 (11h52m53.54903806s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:19.033947286 (11h52m53.549049873s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:18.989442137 (11h53m53.504544716s) [upload=12319]: Next segment err: context deadline exceeded Aug 7 21:33:18.989483135 (11h53m53.504585705s) [upload=12319]: Done uploading segments... Aug 7 21:33:18.989519237 (11h53m53.504621808s) [upload=12319]: Upload finished. err: context deadline exceeded Aug 7 21:33:19.125188732 (11h53m53.640291297s) [upload=12333]: Starting upload Aug 7 21:33:19.125248506 (11h53m53.640351083s) [upload=12333]: Done starting upload Aug 7 21:33:19.125315309 (11h53m53.640417872s) [upload=12333]: Uploading segments... Aug 7 21:33:19.125516847 (11h53m53.640619416s) [upload=12333]: Got next segment. Inline: false Aug 7 21:33:19.125552445 (11h53m53.640655011s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:19.125557201 (11h53m53.640659768s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:19.079113014 (11h54m53.594215592s) [upload=12333]: Next segment err: context deadline exceeded Aug 7 21:34:19.079283700 (11h54m53.594386266s) [upload=12333]: Done uploading segments... Aug 7 21:34:19.079328529 (11h54m53.594431106s) [upload=12333]: Upload finished. err: context deadline exceeded Aug 7 21:34:19.187567059 (11h54m53.702669626s) [upload=12350]: Starting upload Aug 7 21:34:19.187648155 (11h54m53.702750831s) [upload=12350]: Done starting upload Aug 7 21:34:19.187973975 (11h54m53.703076544s) [upload=12350]: Uploading segments... Aug 7 21:34:19.188029730 (11h54m53.703132313s) [upload=12350]: Got next segment. Inline: false Aug 7 21:34:19.188059462 (11h54m53.703162028s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.188074521 (11h54m53.703177098s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:19.148941011 (11h55m53.664043596s) [upload=12350]: Next segment err: context deadline exceeded Aug 7 21:35:19.149031518 (11h55m53.664134089s) [upload=12350]: Done uploading segments... Aug 7 21:35:19.149047732 (11h55m53.664150299s) [upload=12350]: Upload finished. err: context deadline exceeded Aug 7 21:35:19.273234768 (11h55m53.788337346s) [upload=12365]: Starting upload Aug 7 21:35:19.273323516 (11h55m53.788426094s) [upload=12365]: Done starting upload Aug 7 21:35:19.273394181 (11h55m53.788496786s) [upload=12365]: Uploading segments... Aug 7 21:35:19.273671406 (11h55m53.78877401s) [upload=12365]: Got next segment. Inline: false Aug 7 21:35:19.273763286 (11h55m53.788865849s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:19.273769994 (11h55m53.788872683s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:36:19.240552666 (11h56m53.755655246s) [upload=12365]: Next segment err: context deadline exceeded Aug 7 21:36:19.240671719 (11h56m53.755774299s) [upload=12365]: Done uploading segments... Aug 7 21:36:19.240701389 (11h56m53.755803967s) [upload=12365]: Upload finished. err: context deadline exceeded Aug 7 21:36:19.345318455 (11h56m53.860421018s) [upload=12380]: Starting upload Aug 7 21:36:19.345425351 (11h56m53.860527928s) [upload=12380]: Done starting upload Aug 7 21:36:19.345535518 (11h56m53.860638087s) [upload=12380]: Uploading segments... Aug 7 21:36:19.346003455 (11h56m53.861106021s) [upload=12380]: Got next segment. Inline: false Aug 7 21:36:19.346050328 (11h56m53.861152892s) [upload=12380]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:19.346072572 (11h56m53.861175152s) [upload=12380]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:37:19.314409078 (11h57m53.82951166s) [upload=12380]: Next segment err: context deadline exceeded Aug 7 21:37:19.314490160 (11h57m53.82959273s) [upload=12380]: Done uploading segments... Aug 7 21:37:19.314514603 (11h57m53.829617171s) [upload=12380]: Upload finished. err: context deadline exceeded Aug 7 21:37:19.426411282 (11h57m53.941513863s) [upload=12393]: Starting upload Aug 7 21:37:19.426476243 (11h57m53.941578826s) [upload=12393]: Done starting upload Aug 7 21:37:19.426815434 (11h57m53.941917998s) [upload=12393]: Uploading segments... Aug 7 21:37:19.426871686 (11h57m53.941974272s) [upload=12393]: Got next segment. Inline: false Aug 7 21:37:19.426910721 (11h57m53.942013293s) [upload=12393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:19.426916766 (11h57m53.942019353s) [upload=12393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:38:19.396057191 (11h58m53.911159771s) [upload=12393]: Next segment err: context deadline exceeded Aug 7 21:38:19.396186247 (11h58m53.911288816s) [upload=12393]: Done uploading segments... Aug 7 21:38:19.396226941 (11h58m53.911329511s) [upload=12393]: Upload finished. err: context deadline exceeded Aug 7 21:38:29.817879457 (11h59m4.332982023s) [upload=12410]: Starting upload Aug 7 21:38:29.817966266 (11h59m4.333068848s) [upload=12410]: Done starting upload Aug 7 21:38:29.818043990 (11h59m4.333146567s) [upload=12410]: Uploading segments... Aug 7 21:38:29.818242714 (11h59m4.333345283s) [upload=12410]: Got next segment. Inline: false Aug 7 21:38:29.818287021 (11h59m4.333389605s) [upload=12410]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:29.818295454 (11h59m4.333398035s) [upload=12410]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:39:29.785012356 (12h0m4.300114939s) [upload=12410]: Next segment err: context deadline exceeded Aug 7 21:39:29.785073992 (12h0m4.300176561s) [upload=12410]: Done uploading segments... Aug 7 21:39:29.785090740 (12h0m4.30019331s) [upload=12410]: Upload finished. err: context deadline exceeded
Duration: 1m0.000892083s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000713866s
FAIL (2025-08-07 21:39:21 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000736839s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
276.015469ms
FAIL (2025-08-07 21:39:19 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests
Duration: 276.065985ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
34 out of 60 is failed (threshold=3) [........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.00095809s
FAIL (2025-08-07 21:39:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00097207s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
32 out of 60 is failed (threshold=3) [...........................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001034699s
FAIL (2025-08-07 21:39:03 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.464264947 (11h59m7.979367529s) [upload=12412]: Starting upload Aug 7 21:38:33.464398836 (11h59m7.97950143s) [upload=12412]: Done starting upload Aug 7 21:38:33.464523038 (11h59m7.979625617s) [upload=12412]: Uploading segments... Aug 7 21:38:33.464582712 (11h59m7.979685297s) [upload=12412]: Got next segment. Inline: true Aug 7 21:38:33.464590557 (11h59m7.979693146s) [upload=12412]: Waiting for error group managing segments... Aug 7 21:38:33.464707263 (11h59m7.979809843s) [upload=12412]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.464714091 (11h59m7.979816675s) [upload=12412]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.464718861 (11h59m7.979821444s) [upload=12412]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:39:03.416936873 (11h59m37.932039458s) [upload=12412]: Done uploading segments... Aug 7 21:39:03.416957789 (11h59m37.932060367s) [upload=12412]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001085547s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001136408s
FAIL (2025-08-07 21:38:51 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:51.654454416 (11h58m26.169556984s) [upload=12402]: Starting upload Aug 7 21:37:51.654520098 (11h58m26.169622676s) [upload=12402]: Done starting upload Aug 7 21:37:51.654614276 (11h58m26.169716851s) [upload=12402]: Uploading segments... Aug 7 21:37:51.654664780 (11h58m26.169767365s) [upload=12402]: Got next segment. Inline: true Aug 7 21:37:51.654670716 (11h58m26.169773293s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:37:51.654714345 (11h58m26.169816913s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:51.654718808 (11h58m26.169821373s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:51.654721980 (11h58m26.169824545s) [upload=12402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:51.617054052 (11h59m26.132156636s) [upload=12402]: Done uploading segments... Aug 7 21:38:51.617080024 (11h59m26.132182612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001153369s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000841724s
FAIL (2025-08-07 21:38:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000868111s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.000672142s
FAIL (2025-08-07 21:38:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00068505s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
7 out of 60 is failed (threshold=3) [......................................XXX........XX..XX.....] duration:
58.966486ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:38:44 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 59.002265ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
33 out of 60 is failed (threshold=3) [.........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.003543473s
FAIL (2025-08-07 21:38:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.003570939s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001267274s
FAIL (2025-08-07 21:38:37 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ad=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:35:28.227195655 (11h56m2.742298223s) [upload=12367]: Starting upload Aug 7 21:35:28.227255087 (11h56m2.742357657s) [upload=12367]: Done starting upload Aug 7 21:35:28.227336170 (11h56m2.742438737s) [upload=12367]: Uploading segments... Aug 7 21:35:28.227548036 (11h56m2.742650613s) [upload=12367]: Got next segment. Inline: false Aug 7 21:35:28.227579186 (11h56m2.742681753s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:28.227583627 (11h56m2.742686203s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:58.195736071 (11h56m32.710838658s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:35:58.195810513 (11h56m32.710913091s) [upload=12367]: Done uploading segments... Aug 7 21:35:58.195829651 (11h56m32.710932225s) [upload=12367]: Upload finished. err: context deadline exceeded Aug 7 21:36:37.527994350 (11h57m12.043096926s) [upload=12382]: Starting upload Aug 7 21:36:37.528079972 (11h57m12.043182536s) [upload=12382]: Done starting upload Aug 7 21:36:37.528185695 (11h57m12.043288268s) [upload=12382]: Uploading segments... Aug 7 21:36:37.528463728 (11h57m12.043566306s) [upload=12382]: Got next segment. Inline: false Aug 7 21:36:37.528594446 (11h57m12.043697027s) [upload=12382]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:37.528601324 (11h57m12.043703906s) [upload=12382]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:37:07.486412463 (11h57m42.001515034s) [upload=12382]: Next segment err: context deadline exceeded Aug 7 21:37:07.486493636 (11h57m42.001596204s) [upload=12382]: Done uploading segments... Aug 7 21:37:07.486510871 (11h57m42.001613436s) [upload=12382]: Upload finished. err: context deadline exceeded Aug 7 21:37:34.078813217 (11h58m8.593915791s) [upload=12397]: Starting upload Aug 7 21:37:34.078881152 (11h58m8.59398373s) [upload=12397]: Done starting upload Aug 7 21:37:34.079073995 (11h58m8.594176557s) [upload=12397]: Uploading segments... Aug 7 21:37:34.080463095 (11h58m8.59556567s) [upload=12397]: Got next segment. Inline: false Aug 7 21:37:34.080601937 (11h58m8.595704517s) [upload=12397]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.080608666 (11h58m8.595711243s) [upload=12397]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:38:04.048079311 (11h58m38.563181877s) [upload=12397]: Next segment err: context deadline exceeded Aug 7 21:38:04.048155743 (11h58m38.56325832s) [upload=12397]: Done uploading segments... Aug 7 21:38:04.048174465 (11h58m38.563277044s) [upload=12397]: Upload finished. err: context deadline exceeded Aug 7 21:38:07.629401418 (11h58m42.144503984s) [upload=12406]: Starting upload Aug 7 21:38:07.629479055 (11h58m42.144581624s) [upload=12406]: Done starting upload Aug 7 21:38:07.629571988 (11h58m42.144674564s) [upload=12406]: Uploading segments... Aug 7 21:38:07.629900317 (11h58m42.145002894s) [upload=12406]: Got next segment. Inline: false Aug 7 21:38:07.629953043 (11h58m42.145055624s) [upload=12406]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:38:07.629965392 (11h58m42.145067979s) [upload=12406]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:38:37.597200052 (11h59m12.112302638s) [upload=12406]: Next segment err: context deadline exceeded Aug 7 21:38:37.597290448 (11h59m12.112393017s) [upload=12406]: Done uploading segments... Aug 7 21:38:37.597308037 (11h59m12.112410608s) [upload=12406]: Upload finished. err: context deadline exceeded
Duration: 30.001285364s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000596413s
FAIL (2025-08-07 21:38:21 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000619115s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration:
1m0.000650714s
FAIL (2025-08-07 21:38:19 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
iecestore close: conn closed: op
Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments...
Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload
Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload
Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments...
Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false
Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded
Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments...
Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded
Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload
Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload
Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments...
Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false
Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded
Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments...
Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded
Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload
Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload
Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments...
Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false
Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded
Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments...
Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded
Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload
Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload
Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments...
Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false
Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded
Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments...
Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded
Aug 7 21:32:19.033554133 (11h52m53.5486567s) [upload=12319]: Starting upload
Aug 7 21:32:19.033620836 (11h52m53.548723412s) [upload=12319]: Done starting upload
Aug 7 21:32:19.033708593 (11h52m53.548811183s) [upload=12319]: Uploading segments...
Aug 7 21:32:19.033902998 (11h52m53.549005564s) [upload=12319]: Got next segment. Inline: false
Aug 7 21:32:19.033935489 (11h52m53.54903806s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:32:19.033947286 (11h52m53.549049873s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:33:18.989442137 (11h53m53.504544716s) [upload=12319]: Next segment err: context deadline exceeded
Aug 7 21:33:18.989483135 (11h53m53.504585705s) [upload=12319]: Done uploading segments...
Aug 7 21:33:18.989519237 (11h53m53.504621808s) [upload=12319]: Upload finished. err: context deadline exceeded
Aug 7 21:33:19.125188732 (11h53m53.640291297s) [upload=12333]: Starting upload
Aug 7 21:33:19.125248506 (11h53m53.640351083s) [upload=12333]: Done starting upload
Aug 7 21:33:19.125315309 (11h53m53.640417872s) [upload=12333]: Uploading segments...
Aug 7 21:33:19.125516847 (11h53m53.640619416s) [upload=12333]: Got next segment. Inline: false
Aug 7 21:33:19.125552445 (11h53m53.640655011s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:33:19.125557201 (11h53m53.640659768s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:19.079113014 (11h54m53.594215592s) [upload=12333]: Next segment err: context deadline exceeded
Aug 7 21:34:19.079283700 (11h54m53.594386266s) [upload=12333]: Done uploading segments...
Aug 7 21:34:19.079328529 (11h54m53.594431106s) [upload=12333]: Upload finished. err: context deadline exceeded
Aug 7 21:34:19.187567059 (11h54m53.702669626s) [upload=12350]: Starting upload
Aug 7 21:34:19.187648155 (11h54m53.702750831s) [upload=12350]: Done starting upload
Aug 7 21:34:19.187973975 (11h54m53.703076544s) [upload=12350]: Uploading segments...
Aug 7 21:34:19.188029730 (11h54m53.703132313s) [upload=12350]: Got next segment. Inline: false
Aug 7 21:34:19.188059462 (11h54m53.703162028s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:19.188074521 (11h54m53.703177098s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:35:19.148941011 (11h55m53.664043596s) [upload=12350]: Next segment err: context deadline exceeded
Aug 7 21:35:19.149031518 (11h55m53.664134089s) [upload=12350]: Done uploading segments...
Aug 7 21:35:19.149047732 (11h55m53.664150299s) [upload=12350]: Upload finished. err: context deadline exceeded
Aug 7 21:35:19.273234768 (11h55m53.788337346s) [upload=12365]: Starting upload
Aug 7 21:35:19.273323516 (11h55m53.788426094s) [upload=12365]: Done starting upload
Aug 7 21:35:19.273394181 (11h55m53.788496786s) [upload=12365]: Uploading segments...
Aug 7 21:35:19.273671406 (11h55m53.78877401s) [upload=12365]: Got next segment. Inline: false
Aug 7 21:35:19.273763286 (11h55m53.788865849s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:35:19.273769994 (11h55m53.788872683s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:36:19.240552666 (11h56m53.755655246s) [upload=12365]: Next segment err: context deadline exceeded
Aug 7 21:36:19.240671719 (11h56m53.755774299s) [upload=12365]: Done uploading segments...
Aug 7 21:36:19.240701389 (11h56m53.755803967s) [upload=12365]: Upload finished. err: context deadline exceeded
Aug 7 21:36:19.345318455 (11h56m53.860421018s) [upload=12380]: Starting upload
Aug 7 21:36:19.345425351 (11h56m53.860527928s) [upload=12380]: Done starting upload
Aug 7 21:36:19.345535518 (11h56m53.860638087s) [upload=12380]: Uploading segments...
Aug 7 21:36:19.346003455 (11h56m53.861106021s) [upload=12380]: Got next segment. Inline: false
Aug 7 21:36:19.346050328 (11h56m53.861152892s) [upload=12380]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:36:19.346072572 (11h56m53.861175152s) [upload=12380]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:37:19.314409078 (11h57m53.82951166s) [upload=12380]: Next segment err: context deadline exceeded
Aug 7 21:37:19.314490160 (11h57m53.82959273s) [upload=12380]: Done uploading segments...
Aug 7 21:37:19.314514603 (11h57m53.829617171s) [upload=12380]: Upload finished. err: context deadline exceeded
Aug 7 21:37:19.426411282 (11h57m53.941513863s) [upload=12393]: Starting upload
Aug 7 21:37:19.426476243 (11h57m53.941578826s) [upload=12393]: Done starting upload
Aug 7 21:37:19.426815434 (11h57m53.941917998s) [upload=12393]: Uploading segments...
Aug 7 21:37:19.426871686 (11h57m53.941974272s) [upload=12393]: Got next segment. Inline: false
Aug 7 21:37:19.426910721 (11h57m53.942013293s) [upload=12393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:37:19.426916766 (11h57m53.942019353s) [upload=12393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:38:19.396057191 (11h58m53.911159771s) [upload=12393]: Next segment err: context deadline exceeded
Aug 7 21:38:19.396186247 (11h58m53.911288816s) [upload=12393]: Done uploading segments...
Aug 7 21:38:19.396226941 (11h58m53.911329511s) [upload=12393]: Upload finished. err: context deadline exceeded
Duration: 1m0.000667461s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
758.365849ms
FAIL (2025-08-07 21:38:19 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests
Duration: 758.395134ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
31 out of 60 is failed (threshold=3) [............................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001104862s
FAIL (2025-08-07 21:38:07 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:37.627889342 (11h58m12.142991925s) [upload=12399]: Starting upload Aug 7 21:37:37.627980313 (11h58m12.143082888s) [upload=12399]: Done starting upload Aug 7 21:37:37.628038508 (11h58m12.143141085s) [upload=12399]: Uploading segments... Aug 7 21:37:37.628120414 (11h58m12.143222999s) [upload=12399]: Got next segment. Inline: true Aug 7 21:37:37.628126589 (11h58m12.14322917s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:37:37.628184375 (11h58m12.143286951s) [upload=12399]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:37.628190085 (11h58m12.143292667s) [upload=12399]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:37.628193561 (11h58m12.143296128s) [upload=12399]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:38:07.596336666 (11h58m42.111439232s) [upload=12399]: Done uploading segments... Aug 7 21:38:07.596388046 (11h58m42.111490624s) [upload=12399]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001160814s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [.............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001550735s
FAIL (2025-08-07 21:38:04 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ad=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:35:28.227195655 (11h56m2.742298223s) [upload=12367]: Starting upload Aug 7 21:35:28.227255087 (11h56m2.742357657s) [upload=12367]: Done starting upload Aug 7 21:35:28.227336170 (11h56m2.742438737s) [upload=12367]: Uploading segments... Aug 7 21:35:28.227548036 (11h56m2.742650613s) [upload=12367]: Got next segment. Inline: false Aug 7 21:35:28.227579186 (11h56m2.742681753s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:28.227583627 (11h56m2.742686203s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:58.195736071 (11h56m32.710838658s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:35:58.195810513 (11h56m32.710913091s) [upload=12367]: Done uploading segments... Aug 7 21:35:58.195829651 (11h56m32.710932225s) [upload=12367]: Upload finished. err: context deadline exceeded Aug 7 21:36:37.527994350 (11h57m12.043096926s) [upload=12382]: Starting upload Aug 7 21:36:37.528079972 (11h57m12.043182536s) [upload=12382]: Done starting upload Aug 7 21:36:37.528185695 (11h57m12.043288268s) [upload=12382]: Uploading segments... Aug 7 21:36:37.528463728 (11h57m12.043566306s) [upload=12382]: Got next segment. Inline: false Aug 7 21:36:37.528594446 (11h57m12.043697027s) [upload=12382]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:37.528601324 (11h57m12.043703906s) [upload=12382]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:37:07.486412463 (11h57m42.001515034s) [upload=12382]: Next segment err: context deadline exceeded Aug 7 21:37:07.486493636 (11h57m42.001596204s) [upload=12382]: Done uploading segments... Aug 7 21:37:07.486510871 (11h57m42.001613436s) [upload=12382]: Upload finished. err: context deadline exceeded Aug 7 21:37:34.078813217 (11h58m8.593915791s) [upload=12397]: Starting upload Aug 7 21:37:34.078881152 (11h58m8.59398373s) [upload=12397]: Done starting upload Aug 7 21:37:34.079073995 (11h58m8.594176557s) [upload=12397]: Uploading segments... Aug 7 21:37:34.080463095 (11h58m8.59556567s) [upload=12397]: Got next segment. Inline: false Aug 7 21:37:34.080601937 (11h58m8.595704517s) [upload=12397]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.080608666 (11h58m8.595711243s) [upload=12397]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:38:04.048079311 (11h58m38.563181877s) [upload=12397]: Next segment err: context deadline exceeded Aug 7 21:38:04.048155743 (11h58m38.56325832s) [upload=12397]: Done uploading segments... Aug 7 21:38:04.048174465 (11h58m38.563277044s) [upload=12397]: Upload finished. err: context deadline exceeded
Duration: 30.001568997s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
211.610752ms
FAIL (2025-08-07 21:38:03 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests
Duration: 211.63761ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000568423s
FAIL (2025-08-07 21:37:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000581715s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X.........XX......] duration:
52.063303ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:37:53 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 52.087513ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
7 out of 60 is failed (threshold=3) [.......................................XXX........XX..XX....] duration:
46.586373ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:37:52 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 46.61682ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001091028s
FAIL (2025-08-07 21:37:51 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 76s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:51.563221016 (11h57m26.078323582s) [upload=12385]: Starting upload Aug 7 21:36:51.563298485 (11h57m26.078401061s) [upload=12385]: Done starting upload Aug 7 21:36:51.563359392 (11h57m26.07846196s) [upload=12385]: Uploading segments... Aug 7 21:36:51.563399212 (11h57m26.078501798s) [upload=12385]: Got next segment. Inline: true Aug 7 21:36:51.563403588 (11h57m26.078506155s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:36:51.563444462 (11h57m26.07854703s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:51.563448839 (11h57m26.078551417s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:51.563452132 (11h57m26.078554708s) [upload=12385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.529737798 (11h58m26.044840375s) [upload=12385]: Done uploading segments... Aug 7 21:37:51.529763827 (11h58m26.044866403s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001107136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001587299s
FAIL (2025-08-07 21:37:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001603767s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X.........XX.....] duration:
49.913351ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:37:36 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 49.940637ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
30 out of 60 is failed (threshold=3) [.............................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.007839461s
FAIL (2025-08-07 21:37:33 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:03.580821406 (11h57m38.095923994s) [upload=12390]: Starting upload Aug 7 21:37:03.580911249 (11h57m38.096013833s) [upload=12390]: Done starting upload Aug 7 21:37:03.580985164 (11h57m38.09608773s) [upload=12390]: Uploading segments... Aug 7 21:37:03.581033628 (11h57m38.096136216s) [upload=12390]: Got next segment. Inline: true Aug 7 21:37:03.581039403 (11h57m38.096141968s) [upload=12390]: Waiting for error group managing segments... Aug 7 21:37:03.581095949 (11h57m38.096198512s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:37:03.581101049 (11h57m38.096203614s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:03.581105246 (11h57m38.096207813s) [upload=12390]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:37:33.548640041 (11h58m8.063742605s) [upload=12390]: Done uploading segments... Aug 7 21:37:33.548657711 (11h58m8.063760276s) [upload=12390]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.007869681s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
32 out of 60 is failed (threshold=3) [..........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000503418s
FAIL (2025-08-07 21:37:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000522468s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001602775s
FAIL (2025-08-07 21:37:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001615201s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration:
1m0.000543074s
FAIL (2025-08-07 21:37:19 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
, address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments...
Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload
Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload
Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments...
Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false
Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded
Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments...
Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded
Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload
Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload
Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments...
Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false
Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded
Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments...
Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded
Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload
Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload
Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments...
Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false
Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded
Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments...
Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded
Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload
Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload
Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments...
Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false
Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded
Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments...
Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded
Aug 7 21:32:19.033554133 (11h52m53.5486567s) [upload=12319]: Starting upload
Aug 7 21:32:19.033620836 (11h52m53.548723412s) [upload=12319]: Done starting upload
Aug 7 21:32:19.033708593 (11h52m53.548811183s) [upload=12319]: Uploading segments...
Aug 7 21:32:19.033902998 (11h52m53.549005564s) [upload=12319]: Got next segment. Inline: false
Aug 7 21:32:19.033935489 (11h52m53.54903806s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:32:19.033947286 (11h52m53.549049873s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:33:18.989442137 (11h53m53.504544716s) [upload=12319]: Next segment err: context deadline exceeded
Aug 7 21:33:18.989483135 (11h53m53.504585705s) [upload=12319]: Done uploading segments...
Aug 7 21:33:18.989519237 (11h53m53.504621808s) [upload=12319]: Upload finished. err: context deadline exceeded
Aug 7 21:33:19.125188732 (11h53m53.640291297s) [upload=12333]: Starting upload
Aug 7 21:33:19.125248506 (11h53m53.640351083s) [upload=12333]: Done starting upload
Aug 7 21:33:19.125315309 (11h53m53.640417872s) [upload=12333]: Uploading segments...
Aug 7 21:33:19.125516847 (11h53m53.640619416s) [upload=12333]: Got next segment. Inline: false
Aug 7 21:33:19.125552445 (11h53m53.640655011s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:33:19.125557201 (11h53m53.640659768s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:19.079113014 (11h54m53.594215592s) [upload=12333]: Next segment err: context deadline exceeded
Aug 7 21:34:19.079283700 (11h54m53.594386266s) [upload=12333]: Done uploading segments...
Aug 7 21:34:19.079328529 (11h54m53.594431106s) [upload=12333]: Upload finished. err: context deadline exceeded
Aug 7 21:34:19.187567059 (11h54m53.702669626s) [upload=12350]: Starting upload
Aug 7 21:34:19.187648155 (11h54m53.702750831s) [upload=12350]: Done starting upload
Aug 7 21:34:19.187973975 (11h54m53.703076544s) [upload=12350]: Uploading segments...
Aug 7 21:34:19.188029730 (11h54m53.703132313s) [upload=12350]: Got next segment. Inline: false
Aug 7 21:34:19.188059462 (11h54m53.703162028s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:19.188074521 (11h54m53.703177098s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:35:19.148941011 (11h55m53.664043596s) [upload=12350]: Next segment err: context deadline exceeded
Aug 7 21:35:19.149031518 (11h55m53.664134089s) [upload=12350]: Done uploading segments...
Aug 7 21:35:19.149047732 (11h55m53.664150299s) [upload=12350]: Upload finished. err: context deadline exceeded
Aug 7 21:35:19.273234768 (11h55m53.788337346s) [upload=12365]: Starting upload
Aug 7 21:35:19.273323516 (11h55m53.788426094s) [upload=12365]: Done starting upload
Aug 7 21:35:19.273394181 (11h55m53.788496786s) [upload=12365]: Uploading segments...
Aug 7 21:35:19.273671406 (11h55m53.78877401s) [upload=12365]: Got next segment. Inline: false
Aug 7 21:35:19.273763286 (11h55m53.788865849s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:35:19.273769994 (11h55m53.788872683s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:36:19.240552666 (11h56m53.755655246s) [upload=12365]: Next segment err: context deadline exceeded
Aug 7 21:36:19.240671719 (11h56m53.755774299s) [upload=12365]: Done uploading segments...
Aug 7 21:36:19.240701389 (11h56m53.755803967s) [upload=12365]: Upload finished. err: context deadline exceeded
Aug 7 21:36:19.345318455 (11h56m53.860421018s) [upload=12380]: Starting upload
Aug 7 21:36:19.345425351 (11h56m53.860527928s) [upload=12380]: Done starting upload
Aug 7 21:36:19.345535518 (11h56m53.860638087s) [upload=12380]: Uploading segments...
Aug 7 21:36:19.346003455 (11h56m53.861106021s) [upload=12380]: Got next segment. Inline: false
Aug 7 21:36:19.346050328 (11h56m53.861152892s) [upload=12380]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:36:19.346072572 (11h56m53.861175152s) [upload=12380]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:37:19.314409078 (11h57m53.82951166s) [upload=12380]: Next segment err: context deadline exceeded
Aug 7 21:37:19.314490160 (11h57m53.82959273s) [upload=12380]: Done uploading segments...
Aug 7 21:37:19.314514603 (11h57m53.829617171s) [upload=12380]: Upload finished. err: context deadline exceeded
Duration: 1m0.000557567s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
7 out of 60 is failed (threshold=3) [........................................XXX........XX..XX...] duration:
38.240628ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:37:15 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 38.265811ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000606591s
FAIL (2025-08-07 21:37:10 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000623083s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [..............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000593809s
FAIL (2025-08-07 21:37:07 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:35:28.227195655 (11h56m2.742298223s) [upload=12367]: Starting upload Aug 7 21:35:28.227255087 (11h56m2.742357657s) [upload=12367]: Done starting upload Aug 7 21:35:28.227336170 (11h56m2.742438737s) [upload=12367]: Uploading segments... Aug 7 21:35:28.227548036 (11h56m2.742650613s) [upload=12367]: Got next segment. Inline: false Aug 7 21:35:28.227579186 (11h56m2.742681753s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:28.227583627 (11h56m2.742686203s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:58.195736071 (11h56m32.710838658s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:35:58.195810513 (11h56m32.710913091s) [upload=12367]: Done uploading segments... Aug 7 21:35:58.195829651 (11h56m32.710932225s) [upload=12367]: Upload finished. err: context deadline exceeded Aug 7 21:36:37.527994350 (11h57m12.043096926s) [upload=12382]: Starting upload Aug 7 21:36:37.528079972 (11h57m12.043182536s) [upload=12382]: Done starting upload Aug 7 21:36:37.528185695 (11h57m12.043288268s) [upload=12382]: Uploading segments... Aug 7 21:36:37.528463728 (11h57m12.043566306s) [upload=12382]: Got next segment. Inline: false Aug 7 21:36:37.528594446 (11h57m12.043697027s) [upload=12382]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:36:37.528601324 (11h57m12.043703906s) [upload=12382]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:37:07.486412463 (11h57m42.001515034s) [upload=12382]: Next segment err: context deadline exceeded Aug 7 21:37:07.486493636 (11h57m42.001596204s) [upload=12382]: Done uploading segments... Aug 7 21:37:07.486510871 (11h57m42.001613436s) [upload=12382]: Upload finished. err: context deadline exceeded
Duration: 30.000616073s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
192.245184ms
FAIL (2025-08-07 21:36:54 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests
Duration: 192.269623ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001225907s
FAIL (2025-08-07 21:36:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001742395s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.000775437s
FAIL (2025-08-07 21:36:51 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:51.494104017 (11h56m26.009206585s) [upload=12372]: Starting upload Aug 7 21:35:51.494158922 (11h56m26.009261491s) [upload=12372]: Done starting upload Aug 7 21:35:51.494231720 (11h56m26.009334303s) [upload=12372]: Uploading segments... Aug 7 21:35:51.494274008 (11h56m26.009376593s) [upload=12372]: Got next segment. Inline: true Aug 7 21:35:51.494278672 (11h56m26.009381239s) [upload=12372]: Waiting for error group managing segments... Aug 7 21:35:51.494321240 (11h56m26.009423805s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:51.494325798 (11h56m26.009428378s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:51.494328966 (11h56m26.009431542s) [upload=12372]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:51.456973212 (11h57m25.972075794s) [upload=12372]: Done uploading segments... Aug 7 21:36:51.456996123 (11h57m25.972098701s) [upload=12372]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000805076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.000688664s
FAIL (2025-08-07 21:36:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000702346s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
7 out of 60 is failed (threshold=3) [.........................................XXX........XX..XX..] duration:
94.750531ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:36:44 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 94.773498ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000527965s
FAIL (2025-08-07 21:36:40 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000596531s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
31 out of 60 is failed (threshold=3) [...........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000957209s
FAIL (2025-08-07 21:36:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000975136s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
29 out of 60 is failed (threshold=3) [..............................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000877035s
FAIL (2025-08-07 21:36:23 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 159549 (11h49m34.660262117s) [upload=12267]: Uploading segments... Aug 7 21:29:00.145202218 (11h49m34.660304803s) [upload=12267]: Got next segment. Inline: true Aug 7 21:29:00.145207117 (11h49m34.660309694s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:29:00.145265138 (11h49m34.660367708s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:00.145269988 (11h49m34.660372565s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:00.145273448 (11h49m34.660376024s) [upload=12267]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:30.102071659 (11h50m4.617174235s) [upload=12267]: Done uploading segments... Aug 7 21:29:30.102090179 (11h50m4.617192755s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:33.539143465 (11h50m8.054246041s) [upload=12277]: Starting upload Aug 7 21:29:33.539213325 (11h50m8.054315906s) [upload=12277]: Done starting upload Aug 7 21:29:33.539302991 (11h50m8.054405569s) [upload=12277]: Uploading segments... Aug 7 21:29:33.539365708 (11h50m8.054468296s) [upload=12277]: Got next segment. Inline: true Aug 7 21:29:33.539371692 (11h50m8.054474271s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:29:33.539430803 (11h50m8.054533382s) [upload=12277]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:33.539435504 (11h50m8.054538083s) [upload=12277]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:33.539444907 (11h50m8.054547483s) [upload=12277]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:03.511349456 (11h50m38.026452037s) [upload=12277]: Done uploading segments... Aug 7 21:30:03.511374090 (11h50m38.026476668s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:20.569799768 (11h50m55.084902352s) [upload=12287]: Starting upload Aug 7 21:30:20.569948518 (11h50m55.085051083s) [upload=12287]: Done starting upload Aug 7 21:30:20.570029977 (11h50m55.085132554s) [upload=12287]: Uploading segments... Aug 7 21:30:20.570084652 (11h50m55.085187238s) [upload=12287]: Got next segment. Inline: true Aug 7 21:30:20.570090428 (11h50m55.085193012s) [upload=12287]: Waiting for error group managing segments... Aug 7 21:30:20.570141185 (11h50m55.085243762s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:20.570146823 (11h50m55.085249407s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:20.570150089 (11h50m55.085252672s) [upload=12287]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.533060346 (11h51m25.048162933s) [upload=12287]: Done uploading segments... Aug 7 21:30:50.533083845 (11h51m25.048186427s) [upload=12287]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:12.518937042 (11h51m47.03403961s) [upload=12302]: Starting upload Aug 7 21:31:12.519067772 (11h51m47.03417035s) [upload=12302]: Done starting upload Aug 7 21:31:12.519140868 (11h51m47.034243434s) [upload=12302]: Uploading segments... Aug 7 21:31:12.519184171 (11h51m47.034286756s) [upload=12302]: Got next segment. Inline: true Aug 7 21:31:12.519189306 (11h51m47.034291885s) [upload=12302]: Waiting for error group managing segments... Aug 7 21:31:12.519232622 (11h51m47.034335187s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:12.519237549 (11h51m47.034340124s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:12.519240707 (11h51m47.034343282s) [upload=12302]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:42.488796394 (11h52m17.00389897s) [upload=12302]: Done uploading segments... Aug 7 21:31:42.488815995 (11h52m17.003918561s) [upload=12302]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:42.607461777 (11h52m17.122564345s) [upload=12310]: Starting upload Aug 7 21:31:42.607553930 (11h52m17.122656509s) [upload=12310]: Done starting upload Aug 7 21:31:42.607667902 (11h52m17.122770465s) [upload=12310]: Uploading segments... Aug 7 21:31:42.607807568 (11h52m17.122910155s) [upload=12310]: Got next segment. Inline: true Aug 7 21:31:42.607815069 (11h52m17.122917649s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:31:42.607889585 (11h52m17.122992165s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:42.607895306 (11h52m17.122997882s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:42.607898992 (11h52m17.12300157s) [upload=12310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:12.573817298 (11h52m47.088919866s) [upload=12310]: Done uploading segments... Aug 7 21:32:12.573836595 (11h52m47.088939162s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:28.465291340 (11h53m2.980393908s) [upload=12321]: Starting upload Aug 7 21:32:28.465368394 (11h53m2.98047097s) [upload=12321]: Done starting upload Aug 7 21:32:28.465444145 (11h53m2.980546714s) [upload=12321]: Uploading segments... Aug 7 21:32:28.465485779 (11h53m2.980588364s) [upload=12321]: Got next segment. Inline: true Aug 7 21:32:28.465492323 (11h53m2.980594899s) [upload=12321]: Waiting for error group managing segments... Aug 7 21:32:28.465545136 (11h53m2.980647702s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:28.465549485 (11h53m2.980652062s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:28.465552702 (11h53m2.980655279s) [upload=12321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:58.429641253 (11h53m32.944743829s) [upload=12321]: Done uploading segments... Aug 7 21:32:58.429659470 (11h53m32.944762033s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.541750438 (11h54m2.056853004s) [upload=12337]: Starting upload Aug 7 21:33:27.541821888 (11h54m2.056924465s) [upload=12337]: Done starting upload Aug 7 21:33:27.541902510 (11h54m2.057005079s) [upload=12337]: Uploading segments... Aug 7 21:33:27.541956124 (11h54m2.057058706s) [upload=12337]: Got next segment. Inline: true Aug 7 21:33:27.541961349 (11h54m2.057063925s) [upload=12337]: Waiting for error group managing segments... Aug 7 21:33:27.542025235 (11h54m2.057127812s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.542033416 (11h54m2.057135995s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.542036951 (11h54m2.057139528s) [upload=12337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:57.508102054 (11h54m32.023204631s) [upload=12337]: Done uploading segments... Aug 7 21:33:57.508137493 (11h54m32.023240069s) [upload=12337]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:57.640233448 (11h54m32.155336019s) [upload=12346]: Starting upload Aug 7 21:33:57.640361470 (11h54m32.155464058s) [upload=12346]: Done starting upload Aug 7 21:33:57.640435907 (11h54m32.155538472s) [upload=12346]: Uploading segments... Aug 7 21:33:57.640616292 (11h54m32.155718856s) [upload=12346]: Got next segment. Inline: true Aug 7 21:33:57.640625762 (11h54m32.155728346s) [upload=12346]: Waiting for error group managing segments... Aug 7 21:33:57.640704480 (11h54m32.155807045s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:57.640711081 (11h54m32.155813664s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:57.640715194 (11h54m32.155817777s) [upload=12346]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:27.604993500 (11h55m2.120096077s) [upload=12346]: Done uploading segments... Aug 7 21:34:27.605012253 (11h55m2.120114829s) [upload=12346]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:48.927748781 (11h55m23.442851344s) [upload=12356]: Starting upload Aug 7 21:34:48.927850883 (11h55m23.44295346s) [upload=12356]: Done starting upload Aug 7 21:34:48.927934512 (11h55m23.443037082s) [upload=12356]: Uploading segments... Aug 7 21:34:48.927993966 (11h55m23.443096544s) [upload=12356]: Got next segment. Inline: true Aug 7 21:34:48.927999042 (11h55m23.443101609s) [upload=12356]: Waiting for error group managing segments... Aug 7 21:34:48.928037913 (11h55m23.44314048s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:48.928042486 (11h55m23.443145062s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:48.928045662 (11h55m23.443148229s) [upload=12356]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:18.898381091 (11h55m53.413483668s) [upload=12356]: Done uploading segments... Aug 7 21:35:18.898400600 (11h55m53.413503182s) [upload=12356]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:53.371574449 (11h56m27.886677037s) [upload=12373]: Starting upload Aug 7 21:35:53.371655324 (11h56m27.886757912s) [upload=12373]: Done starting upload Aug 7 21:35:53.371730954 (11h56m27.886833534s) [upload=12373]: Uploading segments... Aug 7 21:35:53.371778501 (11h56m27.886881087s) [upload=12373]: Got next segment. Inline: true Aug 7 21:35:53.371784764 (11h56m27.886887352s) [upload=12373]: Waiting for error group managing segments... Aug 7 21:35:53.371842887 (11h56m27.88694545s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:53.371854360 (11h56m27.886956939s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:53.371858807 (11h56m27.886961388s) [upload=12373]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:36:23.340669795 (11h56m57.855772364s) [upload=12373]: Done uploading segments... Aug 7 21:36:23.340702152 (11h56m57.855804718s) [upload=12373]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000898818s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration:
1m0.000740141s
FAIL (2025-08-07 21:36:19 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
X4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF
Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments...
Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload
Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload
Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments...
Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false
Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded
Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments...
Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded
Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload
Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload
Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments...
Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false
Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded
Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments...
Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded
Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload
Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload
Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments...
Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false
Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded
Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments...
Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded
Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload
Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload
Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments...
Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false
Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded
Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments...
Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded
Aug 7 21:32:19.033554133 (11h52m53.5486567s) [upload=12319]: Starting upload
Aug 7 21:32:19.033620836 (11h52m53.548723412s) [upload=12319]: Done starting upload
Aug 7 21:32:19.033708593 (11h52m53.548811183s) [upload=12319]: Uploading segments...
Aug 7 21:32:19.033902998 (11h52m53.549005564s) [upload=12319]: Got next segment. Inline: false
Aug 7 21:32:19.033935489 (11h52m53.54903806s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:32:19.033947286 (11h52m53.549049873s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:33:18.989442137 (11h53m53.504544716s) [upload=12319]: Next segment err: context deadline exceeded
Aug 7 21:33:18.989483135 (11h53m53.504585705s) [upload=12319]: Done uploading segments...
Aug 7 21:33:18.989519237 (11h53m53.504621808s) [upload=12319]: Upload finished. err: context deadline exceeded
Aug 7 21:33:19.125188732 (11h53m53.640291297s) [upload=12333]: Starting upload
Aug 7 21:33:19.125248506 (11h53m53.640351083s) [upload=12333]: Done starting upload
Aug 7 21:33:19.125315309 (11h53m53.640417872s) [upload=12333]: Uploading segments...
Aug 7 21:33:19.125516847 (11h53m53.640619416s) [upload=12333]: Got next segment. Inline: false
Aug 7 21:33:19.125552445 (11h53m53.640655011s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:33:19.125557201 (11h53m53.640659768s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:19.079113014 (11h54m53.594215592s) [upload=12333]: Next segment err: context deadline exceeded
Aug 7 21:34:19.079283700 (11h54m53.594386266s) [upload=12333]: Done uploading segments...
Aug 7 21:34:19.079328529 (11h54m53.594431106s) [upload=12333]: Upload finished. err: context deadline exceeded
Aug 7 21:34:19.187567059 (11h54m53.702669626s) [upload=12350]: Starting upload
Aug 7 21:34:19.187648155 (11h54m53.702750831s) [upload=12350]: Done starting upload
Aug 7 21:34:19.187973975 (11h54m53.703076544s) [upload=12350]: Uploading segments...
Aug 7 21:34:19.188029730 (11h54m53.703132313s) [upload=12350]: Got next segment. Inline: false
Aug 7 21:34:19.188059462 (11h54m53.703162028s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:19.188074521 (11h54m53.703177098s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:35:19.148941011 (11h55m53.664043596s) [upload=12350]: Next segment err: context deadline exceeded
Aug 7 21:35:19.149031518 (11h55m53.664134089s) [upload=12350]: Done uploading segments...
Aug 7 21:35:19.149047732 (11h55m53.664150299s) [upload=12350]: Upload finished. err: context deadline exceeded
Aug 7 21:35:19.273234768 (11h55m53.788337346s) [upload=12365]: Starting upload
Aug 7 21:35:19.273323516 (11h55m53.788426094s) [upload=12365]: Done starting upload
Aug 7 21:35:19.273394181 (11h55m53.788496786s) [upload=12365]: Uploading segments...
Aug 7 21:35:19.273671406 (11h55m53.78877401s) [upload=12365]: Got next segment. Inline: false
Aug 7 21:35:19.273763286 (11h55m53.788865849s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:35:19.273769994 (11h55m53.788872683s) [upload=12365]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:36:19.240552666 (11h56m53.755655246s) [upload=12365]: Next segment err: context deadline exceeded
Aug 7 21:36:19.240671719 (11h56m53.755774299s) [upload=12365]: Done uploading segments...
Aug 7 21:36:19.240701389 (11h56m53.755803967s) [upload=12365]: Upload finished. err: context deadline exceeded
Duration: 1m0.000770223s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000444791s
FAIL (2025-08-07 21:36:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000463357s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
203.075788ms
FAIL (2025-08-07 21:36:04 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 21:27:51.416158748 (11h48m25.931261311s) [upload=12247]: Upload finished. err: <nil> Aug 7 21:28:45.798342371 (11h49m20.313444948s) [upload=12262]: Starting upload Aug 7 21:28:45.798416305 (11h49m20.313518901s) [upload=12262]: Done starting upload Aug 7 21:28:45.798477819 (11h49m20.313580386s) [upload=12262]: Uploading segments... Aug 7 21:28:45.798575825 (11h49m20.313678419s) [upload=12262]: Got next segment. Inline: true Aug 7 21:28:45.798584522 (11h49m20.313687108s) [upload=12262]: Waiting for error group managing segments... Aug 7 21:28:45.798687034 (11h49m20.31378961s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.798694715 (11h49m20.313797295s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.798698486 (11h49m20.313801062s) [upload=12262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:46.186978049 (11h49m20.70208064s) [upload=12262]: Done uploading segments... Aug 7 21:28:46.187002068 (11h49m20.702104638s) [upload=12262]: Upload finished. err: <nil> Aug 7 21:29:16.127488345 (11h49m50.64259091s) [upload=12270]: Starting upload Aug 7 21:29:16.127549889 (11h49m50.642652457s) [upload=12270]: Done starting upload Aug 7 21:29:16.127615824 (11h49m50.642718402s) [upload=12270]: Uploading segments... Aug 7 21:29:16.127667521 (11h49m50.642770106s) [upload=12270]: Got next segment. Inline: true Aug 7 21:29:16.127672601 (11h49m50.642775181s) [upload=12270]: Waiting for error group managing segments... Aug 7 21:29:16.127712729 (11h49m50.642815309s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:16.127716881 (11h49m50.642819458s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:16.127720133 (11h49m50.642822709s) [upload=12270]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:16.847222817 (11h49m51.362325398s) [upload=12270]: Done uploading segments... Aug 7 21:29:16.847246360 (11h49m51.362348938s) [upload=12270]: Upload finished. err: <nil> Aug 7 21:30:18.010837727 (11h50m52.525940302s) [upload=12284]: Starting upload Aug 7 21:30:18.010911175 (11h50m52.526013752s) [upload=12284]: Done starting upload Aug 7 21:30:18.011057905 (11h50m52.526160468s) [upload=12284]: Uploading segments... Aug 7 21:30:18.011133276 (11h50m52.526235861s) [upload=12284]: Got next segment. Inline: true Aug 7 21:30:18.011139710 (11h50m52.526242293s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:30:18.011213673 (11h50m52.526316251s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:18.011219460 (11h50m52.526322037s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:18.011223955 (11h50m52.526326534s) [upload=12284]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.559764365 (11h50m53.074866933s) [upload=12284]: Done uploading segments... Aug 7 21:30:18.559783248 (11h50m53.074885815s) [upload=12284]: Upload finished. err: <nil> Aug 7 21:31:04.611116835 (11h51m39.126219711s) [upload=12300]: Starting upload Aug 7 21:31:04.611192765 (11h51m39.126295346s) [upload=12300]: Done starting upload Aug 7 21:31:04.611260998 (11h51m39.126363575s) [upload=12300]: Uploading segments... Aug 7 21:31:04.611338584 (11h51m39.126441169s) [upload=12300]: Got next segment. Inline: true Aug 7 21:31:04.611345019 (11h51m39.126447596s) [upload=12300]: Waiting for error group managing segments... Aug 7 21:31:04.611401795 (11h51m39.126504362s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:04.611406788 (11h51m39.126509365s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:04.611410057 (11h51m39.126512633s) [upload=12300]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:04.754380309 (11h51m39.269482886s) [upload=12300]: Done uploading segments... Aug 7 21:31:04.754398631 (11h51m39.269501201s) [upload=12300]: Upload finished. err: <nil> Aug 7 21:31:58.359874277 (11h52m32.874976859s) [upload=12315]: Starting upload Aug 7 21:31:58.359943937 (11h52m32.875046514s) [upload=12315]: Done starting upload Aug 7 21:31:58.360016199 (11h52m32.875118761s) [upload=12315]: Uploading segments... Aug 7 21:31:58.360067457 (11h52m32.875170046s) [upload=12315]: Got next segment. Inline: true Aug 7 21:31:58.360073431 (11h52m32.875176015s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:31:58.360122700 (11h52m32.875225281s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:58.360128305 (11h52m32.875230885s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:58.360133695 (11h52m32.875236277s) [upload=12315]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:58.521220098 (11h52m33.036322664s) [upload=12315]: Done uploading segments... Aug 7 21:31:58.521237221 (11h52m33.036339788s) [upload=12315]: Upload finished. err: <nil> Aug 7 21:32:22.686924772 (11h52m57.202027339s) [upload=12320]: Starting upload Aug 7 21:32:22.686975244 (11h52m57.202077821s) [upload=12320]: Done starting upload Aug 7 21:32:22.687071573 (11h52m57.202174139s) [upload=12320]: Uploading segments... Aug 7 21:32:22.687124884 (11h52m57.20222746s) [upload=12320]: Got next segment. Inline: true Aug 7 21:32:22.687130470 (11h52m57.202233047s) [upload=12320]: Waiting for error group managing segments... Aug 7 21:32:22.687180039 (11h52m57.202282605s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:22.687184827 (11h52m57.202287403s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:22.687199840 (11h52m57.202302415s) [upload=12320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:22.752121580 (11h52m57.267224151s) [upload=12320]: Done uploading segments... Aug 7 21:32:22.752142497 (11h52m57.267245076s) [upload=12320]: Upload finished. err: <nil> Aug 7 21:32:51.703150279 (11h53m26.218252856s) [upload=12329]: Starting upload Aug 7 21:32:51.703229887 (11h53m26.218332456s) [upload=12329]: Done starting upload Aug 7 21:32:51.703335344 (11h53m26.21843792s) [upload=12329]: Uploading segments... Aug 7 21:32:51.703391261 (11h53m26.218493848s) [upload=12329]: Got next segment. Inline: true Aug 7 21:32:51.703415160 (11h53m26.218517737s) [upload=12329]: Waiting for error group managing segments... Aug 7 21:32:51.703469875 (11h53m26.218572452s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.703492513 (11h53m26.218595093s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.703497179 (11h53m26.218599746s) [upload=12329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.795090703 (11h53m26.310193281s) [upload=12329]: Done uploading segments... Aug 7 21:32:51.795110987 (11h53m26.310213565s) [upload=12329]: Upload finished. err: <nil> Aug 7 21:33:25.099424440 (11h53m59.614527027s) [upload=12335]: Starting upload Aug 7 21:33:25.099499758 (11h53m59.614602333s) [upload=12335]: Done starting upload Aug 7 21:33:25.099581429 (11h53m59.614683997s) [upload=12335]: Uploading segments... Aug 7 21:33:25.099651772 (11h53m59.614754357s) [upload=12335]: Got next segment. Inline: true Aug 7 21:33:25.099657104 (11h53m59.61475968s) [upload=12335]: Waiting for error group managing segments... Aug 7 21:33:25.099720188 (11h53m59.614822807s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:25.099725753 (11h53m59.61482833s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:25.099729168 (11h53m59.614831745s) [upload=12335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:25.235454224 (11h53m59.750556811s) [upload=12335]: Done uploading segments... Aug 7 21:33:25.235475691 (11h53m59.750578273s) [upload=12335]: Upload finished. err: <nil> Aug 7 21:33:50.139590101 (11h54m24.654692668s) [upload=12342]: Starting upload Aug 7 21:33:50.139629823 (11h54m24.654732408s) [upload=12342]: Done starting upload Aug 7 21:33:50.139704669 (11h54m24.654807238s) [upload=12342]: Uploading segments... Aug 7 21:33:50.139756746 (11h54m24.654859377s) [upload=12342]: Got next segment. Inline: true Aug 7 21:33:50.139761364 (11h54m24.65486394s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:33:50.139818827 (11h54m24.654921409s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:50.139823983 (11h54m24.654926567s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:50.139827483 (11h54m24.65493006s) [upload=12342]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:50.330982283 (11h54m24.846084859s) [upload=12342]: Done uploading segments... Aug 7 21:33:50.330998821 (11h54m24.846101387s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:34:38.241631350 (11h55m12.756733928s) [upload=12353]: Starting upload Aug 7 21:34:38.241732361 (11h55m12.756834946s) [upload=12353]: Done starting upload Aug 7 21:34:38.241861971 (11h55m12.756964555s) [upload=12353]: Uploading segments... Aug 7 21:34:38.241952886 (11h55m12.757055472s) [upload=12353]: Got next segment. Inline: true Aug 7 21:34:38.241959164 (11h55m12.757061725s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:34:38.242038173 (11h55m12.757140761s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:38.242044540 (11h55m12.757147119s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:38.242048944 (11h55m12.757151524s) [upload=12353]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:38.250263994 (11h55m12.765366563s) [upload=12353]: Done uploading segments... Aug 7 21:34:38.250282971 (11h55m12.765385546s) [upload=12353]: Upload finished. err: metaclient: Too Many Requests
Duration: 203.125539ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [...............................X..XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.00117744s
FAIL (2025-08-07 21:35:58 UTC)
operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6]: Next segment err: context deadline exceeded Aug 7 21:27:48.338062332 (11h48m22.853164909s) [upload=12236]: Done uploading segments... Aug 7 21:27:48.338107638 (11h48m22.853210206s) [upload=12236]: Upload finished. err: context deadline exceeded Aug 7 21:28:15.574364584 (11h48m50.08946716s) [upload=12254]: Starting upload Aug 7 21:28:15.574457044 (11h48m50.089559621s) [upload=12254]: Done starting upload Aug 7 21:28:15.574575916 (11h48m50.089678484s) [upload=12254]: Uploading segments... Aug 7 21:28:15.574744256 (11h48m50.089846826s) [upload=12254]: Got next segment. Inline: false Aug 7 21:28:15.574793667 (11h48m50.089896244s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:15.574798428 (11h48m50.089901005s) [upload=12254]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:28:45.539746065 (11h49m20.054848651s) [upload=12254]: Next segment err: context deadline exceeded Aug 7 21:28:45.539812244 (11h49m20.054914811s) [upload=12254]: Done uploading segments... Aug 7 21:28:45.539830868 (11h49m20.054933441s) [upload=12254]: Upload finished. err: context deadline exceeded Aug 7 21:28:54.721015891 (11h49m29.236118476s) [upload=12264]: Starting upload Aug 7 21:28:54.721113647 (11h49m29.236216219s) [upload=12264]: Done starting upload Aug 7 21:28:54.721244895 (11h49m29.236347459s) [upload=12264]: Uploading segments... Aug 7 21:28:54.721543152 (11h49m29.23664573s) [upload=12264]: Got next segment. Inline: false Aug 7 21:28:54.721607181 (11h49m29.236709757s) [upload=12264]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:54.721613099 (11h49m29.236715675s) [upload=12264]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:29:24.689259346 (11h49m59.204361922s) [upload=12264]: Next segment err: context deadline exceeded Aug 7 21:29:24.689391265 (11h49m59.204493841s) [upload=12264]: Done uploading segments... Aug 7 21:29:24.689407952 (11h49m59.204510521s) [upload=12264]: Upload finished. err: context deadline exceeded Aug 7 21:29:24.841358932 (11h49m59.356461502s) [upload=12273]: Starting upload Aug 7 21:29:24.841455568 (11h49m59.356558145s) [upload=12273]: Done starting upload Aug 7 21:29:24.841734059 (11h49m59.356836628s) [upload=12273]: Uploading segments... Aug 7 21:29:24.841794591 (11h49m59.356897176s) [upload=12273]: Got next segment. Inline: false Aug 7 21:29:24.841827466 (11h49m59.356930032s) [upload=12273]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.841843294 (11h49m59.356945873s) [upload=12273]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:29:54.809741795 (11h50m29.324844455s) [upload=12273]: Next segment err: context deadline exceeded Aug 7 21:29:54.809767312 (11h50m29.324869911s) [upload=12273]: Done uploading segments... Aug 7 21:29:54.809791154 (11h50m29.324893716s) [upload=12273]: Upload finished. err: context deadline exceeded Aug 7 21:30:35.331321221 (11h51m9.846423802s) [upload=12293]: Starting upload Aug 7 21:30:35.331442725 (11h51m9.846545308s) [upload=12293]: Done starting upload Aug 7 21:30:35.331555925 (11h51m9.846658503s) [upload=12293]: Uploading segments... Aug 7 21:30:35.331825934 (11h51m9.846928514s) [upload=12293]: Got next segment. Inline: false Aug 7 21:30:35.331908285 (11h51m9.847010862s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:35.331914183 (11h51m9.847016759s) [upload=12293]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:05.292973275 (11h51m39.808075854s) [upload=12293]: Next segment err: context deadline exceeded Aug 7 21:31:05.293128422 (11h51m39.80823099s) [upload=12293]: Done uploading segments... Aug 7 21:31:05.293149038 (11h51m39.808251618s) [upload=12293]: Upload finished. err: context deadline exceeded Aug 7 21:31:07.001756847 (11h51m41.516859446s) [upload=12301]: Starting upload Aug 7 21:31:07.001807635 (11h51m41.516910204s) [upload=12301]: Done starting upload Aug 7 21:31:07.002067943 (11h51m41.517170521s) [upload=12301]: Uploading segments... Aug 7 21:31:07.002110111 (11h51m41.517212693s) [upload=12301]: Got next segment. Inline: false Aug 7 21:31:07.002144574 (11h51m41.517247151s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:07.002148887 (11h51m41.517251463s) [upload=12301]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:31:36.970729876 (11h52m11.485832459s) [upload=12301]: Next segment err: context deadline exceeded Aug 7 21:31:36.970812647 (11h52m11.485915226s) [upload=12301]: Done uploading segments... Aug 7 21:31:36.970834663 (11h52m11.485937233s) [upload=12301]: Upload finished. err: context deadline exceeded Aug 7 21:31:37.097074833 (11h52m11.612177419s) [upload=12309]: Starting upload Aug 7 21:31:37.097210458 (11h52m11.612313036s) [upload=12309]: Done starting upload Aug 7 21:31:37.097377270 (11h52m11.612479844s) [upload=12309]: Uploading segments... Aug 7 21:31:37.097664239 (11h52m11.612766815s) [upload=12309]: Got next segment. Inline: false Aug 7 21:31:37.097738935 (11h52m11.612841503s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:37.097744226 (11h52m11.612846802s) [upload=12309]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:32:07.063322393 (11h52m41.578424976s) [upload=12309]: Next segment err: context deadline exceeded Aug 7 21:32:07.063392855 (11h52m41.578495421s) [upload=12309]: Done uploading segments... Aug 7 21:32:07.063412938 (11h52m41.5785155s) [upload=12309]: Upload finished. err: context deadline exceeded Aug 7 21:32:37.690290086 (11h53m12.205392671s) [upload=12326]: Starting upload Aug 7 21:32:37.690363583 (11h53m12.205466166s) [upload=12326]: Done starting upload Aug 7 21:32:37.690560033 (11h53m12.205662598s) [upload=12326]: Uploading segments... Aug 7 21:32:37.690955738 (11h53m12.206058315s) [upload=12326]: Got next segment. Inline: false Aug 7 21:32:37.690998487 (11h53m12.206101063s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:37.691003164 (11h53m12.20610574s) [upload=12326]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:07.647682616 (11h53m42.162785196s) [upload=12326]: Next segment err: context deadline exceeded Aug 7 21:33:07.647714503 (11h53m42.16281708s) [upload=12326]: Done uploading segments... Aug 7 21:33:07.647729209 (11h53m42.162831776s) [upload=12326]: Upload finished. err: context deadline exceeded Aug 7 21:33:16.586059246 (11h53m51.101161813s) [upload=12332]: Starting upload Aug 7 21:33:16.586135701 (11h53m51.101238278s) [upload=12332]: Done starting upload Aug 7 21:33:16.586241469 (11h53m51.101344046s) [upload=12332]: Uploading segments... Aug 7 21:33:16.586741676 (11h53m51.10184425s) [upload=12332]: Got next segment. Inline: false Aug 7 21:33:16.586822211 (11h53m51.10192479s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:16.586827809 (11h53m51.101930389s) [upload=12332]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:33:46.555115897 (11h54m21.07021848s) [upload=12332]: Next segment err: context deadline exceeded Aug 7 21:33:46.555320570 (11h54m21.070423136s) [upload=12332]: Done uploading segments... Aug 7 21:33:46.555344412 (11h54m21.070446976s) [upload=12332]: Upload finished. err: context deadline exceeded Aug 7 21:34:03.206336186 (11h54m37.721438753s) [upload=12348]: Starting upload Aug 7 21:34:03.206407624 (11h54m37.7215102s) [upload=12348]: Done starting upload Aug 7 21:34:03.206544685 (11h54m37.721647254s) [upload=12348]: Uploading segments... Aug 7 21:34:03.206782614 (11h54m37.72188519s) [upload=12348]: Got next segment. Inline: false Aug 7 21:34:03.206811969 (11h54m37.721914536s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:03.206816355 (11h54m37.721918931s) [upload=12348]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:34:33.174363394 (11h55m7.68946596s) [upload=12348]: Next segment err: context deadline exceeded Aug 7 21:34:33.174465290 (11h55m7.689567866s) [upload=12348]: Done uploading segments... Aug 7 21:34:33.174482682 (11h55m7.689585251s) [upload=12348]: Upload finished. err: context deadline exceeded Aug 7 21:34:42.716392204 (11h55m17.23149477s) [upload=12355]: Starting upload Aug 7 21:34:42.716466463 (11h55m17.231569048s) [upload=12355]: Done starting upload Aug 7 21:34:42.716631833 (11h55m17.231734402s) [upload=12355]: Uploading segments... Aug 7 21:34:42.716905636 (11h55m17.232008212s) [upload=12355]: Got next segment. Inline: false Aug 7 21:34:42.716950835 (11h55m17.232053401s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:42.716955102 (11h55m17.232057682s) [upload=12355]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:12.678206111 (11h55m47.193308708s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:35:12.678280249 (11h55m47.193382812s) [upload=12355]: Done uploading segments... Aug 7 21:35:12.678299114 (11h55m47.193401676s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:35:28.227195655 (11h56m2.742298223s) [upload=12367]: Starting upload Aug 7 21:35:28.227255087 (11h56m2.742357657s) [upload=12367]: Done starting upload Aug 7 21:35:28.227336170 (11h56m2.742438737s) [upload=12367]: Uploading segments... Aug 7 21:35:28.227548036 (11h56m2.742650613s) [upload=12367]: Got next segment. Inline: false Aug 7 21:35:28.227579186 (11h56m2.742681753s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:35:28.227583627 (11h56m2.742686203s) [upload=12367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 7 21:35:58.195736071 (11h56m32.710838658s) [upload=12367]: Next segment err: context deadline exceeded Aug 7 21:35:58.195810513 (11h56m32.710913091s) [upload=12367]: Done uploading segments... Aug 7 21:35:58.195829651 (11h56m32.710932225s) [upload=12367]: Upload finished. err: context deadline exceeded
Duration: 30.001202877s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [.............................................X.........XX...] duration:
31.596927ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:35:55 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 31.697024ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration:
1m0.000791674s
FAIL (2025-08-07 21:35:51 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3s) [upload=12213]: Uploading segments... Aug 7 21:25:50.700993944 (11h46m25.216096531s) [upload=12213]: Got next segment. Inline: true Aug 7 21:25:50.700998564 (11h46m25.216101141s) [upload=12213]: Waiting for error group managing segments... Aug 7 21:25:50.701048517 (11h46m25.216151094s) [upload=12213]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:25:50.701053549 (11h46m25.216156126s) [upload=12213]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:50.701057170 (11h46m25.216159746s) [upload=12213]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:26:50.671344265 (11h47m25.186446841s) [upload=12213]: Done uploading segments... Aug 7 21:26:50.671362828 (11h47m25.186465404s) [upload=12213]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:50.776629761 (11h47m25.291732341s) [upload=12231]: Starting upload Aug 7 21:26:50.776732965 (11h47m25.29183553s) [upload=12231]: Done starting upload Aug 7 21:26:50.776845506 (11h47m25.291948089s) [upload=12231]: Uploading segments... Aug 7 21:26:50.776902866 (11h47m25.292005453s) [upload=12231]: Got next segment. Inline: true Aug 7 21:26:50.776976198 (11h47m25.292078777s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:26:50.777036232 (11h47m25.292138808s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:50.777040463 (11h47m25.292143039s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:50.777045905 (11h47m25.292148481s) [upload=12231]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:27:50.741322246 (11h48m25.256424823s) [upload=12231]: Done uploading segments... Aug 7 21:27:50.741349320 (11h48m25.256451896s) [upload=12231]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:50.868082451 (11h48m25.383185029s) [upload=12246]: Starting upload Aug 7 21:27:50.868168448 (11h48m25.383271036s) [upload=12246]: Done starting upload Aug 7 21:27:50.868259208 (11h48m25.383361776s) [upload=12246]: Uploading segments... Aug 7 21:27:50.868322360 (11h48m25.383424949s) [upload=12246]: Got next segment. Inline: true Aug 7 21:27:50.868327737 (11h48m25.38343033s) [upload=12246]: Waiting for error group managing segments... Aug 7 21:27:50.868372105 (11h48m25.383474674s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:27:50.868390100 (11h48m25.38349268s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:50.868393651 (11h48m25.383496233s) [upload=12246]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:28:50.822345206 (11h49m25.337447778s) [upload=12246]: Done uploading segments... Aug 7 21:28:50.822369534 (11h49m25.337472109s) [upload=12246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:50.930403175 (11h49m25.445505759s) [upload=12263]: Starting upload Aug 7 21:28:50.930542685 (11h49m25.445645273s) [upload=12263]: Done starting upload Aug 7 21:28:50.930631901 (11h49m25.445734464s) [upload=12263]: Uploading segments... Aug 7 21:28:50.930741078 (11h49m25.445843663s) [upload=12263]: Got next segment. Inline: true Aug 7 21:28:50.930747674 (11h49m25.445850257s) [upload=12263]: Waiting for error group managing segments... Aug 7 21:28:50.930820590 (11h49m25.445923176s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:28:50.930826266 (11h49m25.445928855s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:50.930829970 (11h49m25.445932549s) [upload=12263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:29:50.886203407 (11h50m25.401305972s) [upload=12263]: Done uploading segments... Aug 7 21:29:50.886222069 (11h50m25.401324635s) [upload=12263]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:51.009688685 (11h50m25.524791268s) [upload=12280]: Starting upload Aug 7 21:29:51.009815101 (11h50m25.524917674s) [upload=12280]: Done starting upload Aug 7 21:29:51.009946440 (11h50m25.525049009s) [upload=12280]: Uploading segments... Aug 7 21:29:51.010026512 (11h50m25.5251291s) [upload=12280]: Got next segment. Inline: true Aug 7 21:29:51.010047258 (11h50m25.525149838s) [upload=12280]: Waiting for error group managing segments... Aug 7 21:29:51.010094971 (11h50m25.525197557s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:29:51.010099871 (11h50m25.525202451s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:51.010103444 (11h50m25.525206022s) [upload=12280]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:30:50.972236265 (11h51m25.487338842s) [upload=12280]: Done uploading segments... Aug 7 21:30:50.972259798 (11h51m25.487362362s) [upload=12280]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:51.089988464 (11h51m25.605091032s) [upload=12297]: Starting upload Aug 7 21:30:51.090059227 (11h51m25.605161804s) [upload=12297]: Done starting upload Aug 7 21:30:51.090142338 (11h51m25.605244913s) [upload=12297]: Uploading segments... Aug 7 21:30:51.090195293 (11h51m25.60529788s) [upload=12297]: Got next segment. Inline: true Aug 7 21:30:51.090201259 (11h51m25.605303848s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:30:51.090256559 (11h51m25.605359137s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:30:51.090263471 (11h51m25.605366051s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:51.090268093 (11h51m25.605370673s) [upload=12297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:31:51.057826723 (11h52m25.572929299s) [upload=12297]: Done uploading segments... Aug 7 21:31:51.057861179 (11h52m25.572963756s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:51.175530039 (11h52m25.690632622s) [upload=12312]: Starting upload Aug 7 21:31:51.175624413 (11h52m25.690727004s) [upload=12312]: Done starting upload Aug 7 21:31:51.175693740 (11h52m25.690796308s) [upload=12312]: Uploading segments... Aug 7 21:31:51.175770172 (11h52m25.690872757s) [upload=12312]: Got next segment. Inline: true Aug 7 21:31:51.175775796 (11h52m25.690878372s) [upload=12312]: Waiting for error group managing segments... Aug 7 21:31:51.175833361 (11h52m25.690935937s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:31:51.175838635 (11h52m25.690941211s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:51.175842099 (11h52m25.690944675s) [upload=12312]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:32:51.144866904 (11h53m25.659969473s) [upload=12312]: Done uploading segments... Aug 7 21:32:51.144906911 (11h53m25.660009488s) [upload=12312]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:51.256659465 (11h53m25.771762033s) [upload=12328]: Starting upload Aug 7 21:32:51.256730239 (11h53m25.771832821s) [upload=12328]: Done starting upload Aug 7 21:32:51.256811354 (11h53m25.771913921s) [upload=12328]: Uploading segments... Aug 7 21:32:51.256901518 (11h53m25.772004104s) [upload=12328]: Got next segment. Inline: true Aug 7 21:32:51.256907925 (11h53m25.772010502s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:32:51.256971919 (11h53m25.772074483s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:32:51.256976811 (11h53m25.772079387s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:51.256980076 (11h53m25.772082642s) [upload=12328]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:33:51.227391235 (11h54m25.742493825s) [upload=12328]: Done uploading segments... Aug 7 21:33:51.227415234 (11h54m25.742517799s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:51.338110474 (11h54m25.85321305s) [upload=12344]: Starting upload Aug 7 21:33:51.338191699 (11h54m25.853294275s) [upload=12344]: Done starting upload Aug 7 21:33:51.338277324 (11h54m25.853379887s) [upload=12344]: Uploading segments... Aug 7 21:33:51.338344126 (11h54m25.853446711s) [upload=12344]: Got next segment. Inline: true Aug 7 21:33:51.338350126 (11h54m25.853452712s) [upload=12344]: Waiting for error group managing segments... Aug 7 21:33:51.338407582 (11h54m25.853510158s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:33:51.338412570 (11h54m25.853515147s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:51.338441642 (11h54m25.853544221s) [upload=12344]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:34:51.299810563 (11h55m25.814913142s) [upload=12344]: Done uploading segments... Aug 7 21:34:51.299835572 (11h55m25.81493815s) [upload=12344]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:51.407688064 (11h55m25.922790632s) [upload=12357]: Starting upload Aug 7 21:34:51.407849580 (11h55m25.922952149s) [upload=12357]: Done starting upload Aug 7 21:34:51.407998543 (11h55m25.923101112s) [upload=12357]: Uploading segments... Aug 7 21:34:51.408079736 (11h55m25.923182323s) [upload=12357]: Got next segment. Inline: true Aug 7 21:34:51.408088235 (11h55m25.923190826s) [upload=12357]: Waiting for error group managing segments... Aug 7 21:34:51.408154168 (11h55m25.923256744s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:34:51.408159236 (11h55m25.923261813s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:51.408162802 (11h55m25.92326538s) [upload=12357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:35:51.370892367 (11h56m25.88599495s) [upload=12357]: Done uploading segments... Aug 7 21:35:51.370918800 (11h56m25.88602138s) [upload=12357]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000808169s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
7 out of 60 is failed (threshold=3) [..........................................XXX........XX..XX.] duration:
46.64365ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:35:50 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 46.677519ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.001887182s
FAIL (2025-08-07 21:35:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001914837s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
30 out of 60 is failed (threshold=3) [............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001193161s
FAIL (2025-08-07 21:35:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001211529s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.001236207s
FAIL (2025-08-07 21:35:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001264696s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.000569633s
FAIL (2025-08-07 21:35:26 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000596274s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration:
1m0.001415809s
FAIL (2025-08-07 21:35:19 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
BIZVODRKIQQK6JGXAKJVEMQ4BGOP2DOOVKO3CDGA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.754982553 (11h39m30.270085122s) [noise=true , address=104.36.82.9:4023 , piece_id=DVV34YYQ44ELHTF2H5JAQQAXCKXYROYNUCSAFDYAAJRH5JB5I6GQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.763242025 (11h39m30.278344591s) [noise=true , address=104.36.82.9:4014 , piece_id=GT2W2ITZ4L5YJV26Z7TKI43K5OTEW5BDYBMC264ADGBUI6A24AZQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.767097571 (11h39m30.282200149s) [noise=true , address=104.36.82.60:4355 , piece_id=DRT3SUTNHSQN7GIKTVWLFVIUUFYWAOBQ5HFGVYFS7S77PKRE773Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.795229256 (11h39m30.310331824s) [noise=true , address=79.127.220.97:4396 , piece_id=FGUVLR4ZORGKRGPN4J7JC7TIX4OD4HBVDJAGHVTS4X4VFWSOEW3Q, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.946142079 (11h39m30.461244645s) [noise=true , address=104.36.82.16:4299 , piece_id=OVN5ITCXBZ3X42777GE6UZL3MOSONJWG75FQ35HQNS6MO6VWLGHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: <nil>
Aug 7 21:18:55.946161863 (11h39m30.461264432s) [seg_pos={0 0}, upload=12100]: Segment reached optimal threshold of 14 pieces.
Aug 7 21:18:55.946440697 (11h39m30.461543263s) [noise=true , address=104.36.82.54:4191 , piece_id=TCW2FNAM2FX5IQB765MSA6IXRAP2DA4QRIHTJXX35YXDM2QWY6OA, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF
Aug 7 21:18:55.946812303 (11h39m30.461914878s) [noise=true , address=64.49.13.170:4725 , piece_id=4OC6KAXONJQYIYCU5PIYMWUFEEHWIXQRK77UGHJSUKU2VUZNPMHQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mit6F4Yz1qKuVeXhhfBpNd5MsfxWLz8dCUAHsBeGzCdLJn8Ne): protocol: read tcp 159.223.143.39:35190->64.49.13.170:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946882266 (11h39m30.461984836s) [noise=true , address=104.36.82.44:4886 , piece_id=QKRZX4DGSZO2KZS4W6XC2MH6U7NSRTESCO3ATCBD4URJEHNOTEBQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: read tcp 159.223.143.39:53726->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946938175 (11h39m30.462040752s) [noise=true , address=64.49.13.138:4486 , piece_id=POHTN5ROSL5OTO3WAXZMYMZTMWJTTN2DSFNJFZFTPCYBIX6LLZVQ, seg_pos={0 0}, upload=12100]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126i47Q2ngKd79VRwsSQXZn3QTrJfmvZAQdK6yqFWXkCo5ba26g): protocol: read tcp 159.223.143.39:59326->64.49.13.138:4486: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 21:18:55.946950540 (11h39m30.462053108s) [seg_pos={0 0}, upload=12100]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 21:18:55.947076214 (11h39m30.462178791s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:18:55.947080872 (11h39m30.462183448s) [upload=12100]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:27:57.892309182 (11h48m32.407411752s) [upload=12100]: Done uploading segments...
Aug 7 21:27:57.892332157 (11h48m32.407434733s) [upload=12100]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:27:58.003229647 (11h48m32.518332224s) [upload=12249]: Starting upload
Aug 7 21:27:58.003323626 (11h48m32.518426204s) [upload=12249]: Done starting upload
Aug 7 21:27:58.003400107 (11h48m32.518502669s) [upload=12249]: Uploading segments...
Aug 7 21:27:58.003810194 (11h48m32.518912834s) [upload=12249]: Got next segment. Inline: false
Aug 7 21:27:58.003889531 (11h48m32.518992107s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:27:58.003895247 (11h48m32.518997823s) [upload=12249]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:28:57.972353893 (11h49m32.487456481s) [upload=12249]: Next segment err: context deadline exceeded
Aug 7 21:28:57.972582036 (11h49m32.487684605s) [upload=12249]: Done uploading segments...
Aug 7 21:28:57.972599877 (11h49m32.487702441s) [upload=12249]: Upload finished. err: context deadline exceeded
Aug 7 21:29:10.592043729 (11h49m45.107146306s) [upload=12268]: Starting upload
Aug 7 21:29:10.592137086 (11h49m45.10723967s) [upload=12268]: Done starting upload
Aug 7 21:29:10.592249553 (11h49m45.107352121s) [upload=12268]: Uploading segments...
Aug 7 21:29:10.592423657 (11h49m45.107526241s) [upload=12268]: Got next segment. Inline: false
Aug 7 21:29:10.592553738 (11h49m45.107656321s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:29:10.592562645 (11h49m45.107665227s) [upload=12268]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:30:10.554991467 (11h50m45.070094052s) [upload=12268]: Next segment err: context deadline exceeded
Aug 7 21:30:10.555194817 (11h50m45.070297394s) [upload=12268]: Done uploading segments...
Aug 7 21:30:10.555226724 (11h50m45.070329293s) [upload=12268]: Upload finished. err: context deadline exceeded
Aug 7 21:30:18.863445746 (11h50m53.378548325s) [upload=12285]: Starting upload
Aug 7 21:30:18.863535413 (11h50m53.378637992s) [upload=12285]: Done starting upload
Aug 7 21:30:18.863666146 (11h50m53.378768724s) [upload=12285]: Uploading segments...
Aug 7 21:30:18.863839373 (11h50m53.378941935s) [upload=12285]: Got next segment. Inline: false
Aug 7 21:30:18.863876545 (11h50m53.378979107s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:30:18.863881942 (11h50m53.378984518s) [upload=12285]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:31:18.830429074 (11h51m53.345531644s) [upload=12285]: Next segment err: context deadline exceeded
Aug 7 21:31:18.830448036 (11h51m53.345550613s) [upload=12285]: Done uploading segments...
Aug 7 21:31:18.830474300 (11h51m53.345576866s) [upload=12285]: Upload finished. err: context deadline exceeded
Aug 7 21:31:18.972600679 (11h51m53.487703265s) [upload=12304]: Starting upload
Aug 7 21:31:18.972700584 (11h51m53.487803182s) [upload=12304]: Done starting upload
Aug 7 21:31:18.972946355 (11h51m53.488048923s) [upload=12304]: Uploading segments...
Aug 7 21:31:18.973271681 (11h51m53.488374263s) [upload=12304]: Got next segment. Inline: false
Aug 7 21:31:18.973342730 (11h51m53.488445311s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:31:18.973348044 (11h51m53.488450626s) [upload=12304]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:32:18.925211587 (11h52m53.440314165s) [upload=12304]: Next segment err: context deadline exceeded
Aug 7 21:32:18.925381198 (11h52m53.440483776s) [upload=12304]: Done uploading segments...
Aug 7 21:32:18.925404293 (11h52m53.440506857s) [upload=12304]: Upload finished. err: context deadline exceeded
Aug 7 21:32:19.033554133 (11h52m53.5486567s) [upload=12319]: Starting upload
Aug 7 21:32:19.033620836 (11h52m53.548723412s) [upload=12319]: Done starting upload
Aug 7 21:32:19.033708593 (11h52m53.548811183s) [upload=12319]: Uploading segments...
Aug 7 21:32:19.033902998 (11h52m53.549005564s) [upload=12319]: Got next segment. Inline: false
Aug 7 21:32:19.033935489 (11h52m53.54903806s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:32:19.033947286 (11h52m53.549049873s) [upload=12319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:33:18.989442137 (11h53m53.504544716s) [upload=12319]: Next segment err: context deadline exceeded
Aug 7 21:33:18.989483135 (11h53m53.504585705s) [upload=12319]: Done uploading segments...
Aug 7 21:33:18.989519237 (11h53m53.504621808s) [upload=12319]: Upload finished. err: context deadline exceeded
Aug 7 21:33:19.125188732 (11h53m53.640291297s) [upload=12333]: Starting upload
Aug 7 21:33:19.125248506 (11h53m53.640351083s) [upload=12333]: Done starting upload
Aug 7 21:33:19.125315309 (11h53m53.640417872s) [upload=12333]: Uploading segments...
Aug 7 21:33:19.125516847 (11h53m53.640619416s) [upload=12333]: Got next segment. Inline: false
Aug 7 21:33:19.125552445 (11h53m53.640655011s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:33:19.125557201 (11h53m53.640659768s) [upload=12333]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:34:19.079113014 (11h54m53.594215592s) [upload=12333]: Next segment err: context deadline exceeded
Aug 7 21:34:19.079283700 (11h54m53.594386266s) [upload=12333]: Done uploading segments...
Aug 7 21:34:19.079328529 (11h54m53.594431106s) [upload=12333]: Upload finished. err: context deadline exceeded
Aug 7 21:34:19.187567059 (11h54m53.702669626s) [upload=12350]: Starting upload
Aug 7 21:34:19.187648155 (11h54m53.702750831s) [upload=12350]: Done starting upload
Aug 7 21:34:19.187973975 (11h54m53.703076544s) [upload=12350]: Uploading segments...
Aug 7 21:34:19.188029730 (11h54m53.703132313s) [upload=12350]: Got next segment. Inline: false
Aug 7 21:34:19.188059462 (11h54m53.703162028s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:34:19.188074521 (11h54m53.703177098s) [upload=12350]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:35:19.148941011 (11h55m53.664043596s) [upload=12350]: Next segment err: context deadline exceeded
Aug 7 21:35:19.149031518 (11h55m53.664134089s) [upload=12350]: Done uploading segments...
Aug 7 21:35:19.149047732 (11h55m53.664150299s) [upload=12350]: Upload finished. err: context deadline exceeded
Duration: 1m0.001433397s
core
uplink
us-select-1
us1-placement-20
|