Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
16 out of 60 is failed (threshold=3) [...........................................X.XXXXXXXXXXXXXXX] duration:
30.001414708s
FAIL (2025-08-07 21:26:39 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:09.277129109 (11h46m43.792231676s) [upload=12221]: Starting upload Aug 7 21:26:09.277219220 (11h46m43.792321801s) [upload=12221]: Done starting upload Aug 7 21:26:09.277298182 (11h46m43.79240075s) [upload=12221]: Uploading segments... Aug 7 21:26:09.277349333 (11h46m43.792451918s) [upload=12221]: Got next segment. Inline: true Aug 7 21:26:09.277354050 (11h46m43.792456616s) [upload=12221]: Waiting for error group managing segments... Aug 7 21:26:09.277409637 (11h46m43.792512216s) [upload=12221]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:26:09.277414451 (11h46m43.792517028s) [upload=12221]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:09.277418693 (11h46m43.792521274s) [upload=12221]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:26:39.240118866 (11h47m13.755221446s) [upload=12221]: Done uploading segments... Aug 7 21:26:39.240141479 (11h47m13.755244062s) [upload=12221]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001441404s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [............................................X..XXXXXXXXXXXXX] duration:
30.000973265s
FAIL (2025-08-07 21:26:32 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
xt canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments...
Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload
Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload
Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments...
Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false
Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded
Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments...
Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded
Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload
Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload
Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments...
Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false
Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded
Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments...
Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded
Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload
Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload
Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments...
Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false
Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded
Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments...
Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded
Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload
Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload
Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments...
Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false
Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded
Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments...
Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded
Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload
Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload
Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments...
Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false
Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded
Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments...
Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded
Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload
Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload
Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments...
Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false
Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded
Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments...
Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded
Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload
Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload
Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments...
Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false
Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded
Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments...
Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded
Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload
Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload
Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments...
Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false
Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded
Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments...
Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded
Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload
Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload
Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments...
Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false
Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded
Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments...
Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded
Aug 7 21:26:02.893464222 (11h46m37.408566798s) [upload=12219]: Starting upload
Aug 7 21:26:02.893541319 (11h46m37.408643901s) [upload=12219]: Done starting upload
Aug 7 21:26:02.893669935 (11h46m37.408772505s) [upload=12219]: Uploading segments...
Aug 7 21:26:02.894015397 (11h46m37.409117967s) [upload=12219]: Got next segment. Inline: false
Aug 7 21:26:02.894075262 (11h46m37.409177827s) [upload=12219]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:26:02.894080024 (11h46m37.409182599s) [upload=12219]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:26:32.863282599 (11h47m7.378385178s) [upload=12219]: Next segment err: context deadline exceeded
Aug 7 21:26:32.863452265 (11h47m7.378554842s) [upload=12219]: Done uploading segments...
Aug 7 21:26:32.863473782 (11h47m7.378576352s) [upload=12219]: Upload finished. err: context deadline exceeded
Duration: 30.000990082s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration:
30.001697692s
FAIL (2025-08-07 21:26:32 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001723408s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
30.015337ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:26:25 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 30.10526ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration:
30.00084308s
FAIL (2025-08-07 21:26:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000867439s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration:
1m0.000944812s
FAIL (2025-08-07 21:26:20 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000980985s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.............................................X..XXXXXXXXXXXX] duration:
30.001473408s
FAIL (2025-08-07 21:26:02 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ress=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments...
Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload
Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload
Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments...
Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false
Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded
Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments...
Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded
Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload
Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload
Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments...
Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false
Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded
Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments...
Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded
Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload
Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload
Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments...
Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false
Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded
Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments...
Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded
Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload
Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload
Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments...
Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false
Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded
Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments...
Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded
Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload
Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload
Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments...
Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false
Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded
Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments...
Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded
Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload
Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload
Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments...
Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false
Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded
Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments...
Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded
Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload
Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload
Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments...
Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false
Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded
Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments...
Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded
Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload
Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload
Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments...
Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false
Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded
Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments...
Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded
Aug 7 21:25:32.808985270 (11h46m7.324087856s) [upload=12211]: Starting upload
Aug 7 21:25:32.809115921 (11h46m7.324218498s) [upload=12211]: Done starting upload
Aug 7 21:25:32.809246128 (11h46m7.324348696s) [upload=12211]: Uploading segments...
Aug 7 21:25:32.809483559 (11h46m7.324586136s) [upload=12211]: Got next segment. Inline: false
Aug 7 21:25:32.809527234 (11h46m7.324629802s) [upload=12211]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:25:32.809532172 (11h46m7.324634749s) [upload=12211]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:26:02.778343587 (11h46m37.293446174s) [upload=12211]: Next segment err: context deadline exceeded
Aug 7 21:26:02.778368526 (11h46m37.293471108s) [upload=12211]: Done uploading segments...
Aug 7 21:26:02.778403796 (11h46m37.293506372s) [upload=12211]: Upload finished. err: context deadline exceeded
Duration: 30.001496225s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration:
30.000544961s
FAIL (2025-08-07 21:25:54 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000567308s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
15 out of 60 is failed (threshold=3) [............................................X.XXXXXXXXXXXXXX] duration:
30.000675242s
FAIL (2025-08-07 21:25:52 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:22.928698200 (11h45m57.443800776s) [upload=12209]: Starting upload Aug 7 21:25:22.928759864 (11h45m57.44386244s) [upload=12209]: Done starting upload Aug 7 21:25:22.928818830 (11h45m57.443921408s) [upload=12209]: Uploading segments... Aug 7 21:25:22.928859114 (11h45m57.4439617s) [upload=12209]: Got next segment. Inline: true Aug 7 21:25:22.928863388 (11h45m57.443965964s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:25:22.928899099 (11h45m57.444001676s) [upload=12209]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:25:22.928902708 (11h45m57.444005284s) [upload=12209]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:22.928905103 (11h45m57.444007679s) [upload=12209]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:52.897461396 (11h46m27.412563973s) [upload=12209]: Done uploading segments... Aug 7 21:25:52.897485286 (11h46m27.412587867s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.000694478s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration:
1m0.001235474s
FAIL (2025-08-07 21:25:50 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:50.576861942 (11h45m25.091964521s) [upload=12202]: Starting upload Aug 7 21:24:50.576970088 (11h45m25.092072667s) [upload=12202]: Done starting upload Aug 7 21:24:50.577081150 (11h45m25.092183733s) [upload=12202]: Uploading segments... Aug 7 21:24:50.577144915 (11h45m25.092247502s) [upload=12202]: Got next segment. Inline: true Aug 7 21:24:50.577152119 (11h45m25.092254707s) [upload=12202]: Waiting for error group managing segments... Aug 7 21:24:50.577348819 (11h45m25.092451467s) [upload=12202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:50.577373232 (11h45m25.092475819s) [upload=12202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:50.577377473 (11h45m25.092480057s) [upload=12202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:50.547385515 (11h46m25.062488099s) [upload=12202]: Done uploading segments... Aug 7 21:25:50.547410805 (11h46m25.062513381s) [upload=12202]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001256949s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration:
30.00089862s
FAIL (2025-08-07 21:25:47 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000923606s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [.................................................XXX........] duration:
60.648816ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:25:36 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 60.671833ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration:
30.00124038s
FAIL (2025-08-07 21:25:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00126064s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
17 out of 60 is failed (threshold=3) [.........................................X..XXXXXXXXXXXXXXXX] duration:
30.000505786s
FAIL (2025-08-07 21:25:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000536612s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
14 out of 60 is failed (threshold=3) [.............................................X.XXXXXXXXXXXXX] duration:
30.001238875s
FAIL (2025-08-07 21:25:22 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:52.849412599 (11h45m27.364515168s) [upload=12203]: Starting upload Aug 7 21:24:52.849476345 (11h45m27.364578922s) [upload=12203]: Done starting upload Aug 7 21:24:52.849537535 (11h45m27.364640103s) [upload=12203]: Uploading segments... Aug 7 21:24:52.849586048 (11h45m27.364688635s) [upload=12203]: Got next segment. Inline: true Aug 7 21:24:52.849591528 (11h45m27.364694109s) [upload=12203]: Waiting for error group managing segments... Aug 7 21:24:52.849639829 (11h45m27.364742419s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:24:52.849646676 (11h45m27.364749259s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:52.849650596 (11h45m27.364753178s) [upload=12203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:25:22.819457633 (11h45m57.334560218s) [upload=12203]: Done uploading segments... Aug 7 21:25:22.819497033 (11h45m57.334599613s) [upload=12203]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001260753s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
30.882415ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:25:18 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 30.905376ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration:
1m0.001185211s
FAIL (2025-08-07 21:25:13 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001213278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration:
30.001521353s
FAIL (2025-08-07 21:24:59 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.001548315s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..............................................X..XXXXXXXXXXX] duration:
30.000950805s
FAIL (2025-08-07 21:24:57 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
e , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments...
Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload
Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload
Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments...
Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false
Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded
Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments...
Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded
Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload
Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload
Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments...
Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false
Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded
Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments...
Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded
Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload
Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload
Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments...
Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false
Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded
Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments...
Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded
Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload
Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload
Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments...
Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false
Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded
Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments...
Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded
Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload
Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload
Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments...
Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false
Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded
Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments...
Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded
Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload
Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload
Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments...
Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false
Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded
Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments...
Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded
Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload
Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload
Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments...
Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false
Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded
Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments...
Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded
Aug 7 21:24:27.053729819 (11h45m1.568832381s) [upload=12197]: Starting upload
Aug 7 21:24:27.053842899 (11h45m1.568945474s) [upload=12197]: Done starting upload
Aug 7 21:24:27.053981008 (11h45m1.569083577s) [upload=12197]: Uploading segments...
Aug 7 21:24:27.054233975 (11h45m1.569336538s) [upload=12197]: Got next segment. Inline: false
Aug 7 21:24:27.054310067 (11h45m1.569412642s) [upload=12197]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:24:27.054314930 (11h45m1.569417509s) [upload=12197]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:24:57.022771776 (11h45m31.537874352s) [upload=12197]: Next segment err: context deadline exceeded
Aug 7 21:24:57.022853671 (11h45m31.537956241s) [upload=12197]: Done uploading segments...
Aug 7 21:24:57.022871345 (11h45m31.537973912s) [upload=12197]: Upload finished. err: context deadline exceeded
Duration: 30.000966597s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration:
30.001548931s
FAIL (2025-08-07 21:24:52 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001598729s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration:
1m0.000709924s
FAIL (2025-08-07 21:24:50 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 22676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:50.384811510 (11h44m24.8999141s) [upload=12188]: Starting upload Aug 7 21:23:50.384943363 (11h44m24.900045943s) [upload=12188]: Done starting upload Aug 7 21:23:50.385049891 (11h44m24.900152468s) [upload=12188]: Uploading segments... Aug 7 21:23:50.385198215 (11h44m24.900300817s) [upload=12188]: Got next segment. Inline: true Aug 7 21:23:50.385206628 (11h44m24.90030922s) [upload=12188]: Waiting for error group managing segments... Aug 7 21:23:50.385271294 (11h44m24.900373878s) [upload=12188]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:50.385277488 (11h44m24.900380074s) [upload=12188]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:50.385284995 (11h44m24.900387585s) [upload=12188]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.354420606 (11h45m24.869523188s) [upload=12188]: Done uploading segments... Aug 7 21:24:50.354462079 (11h45m24.869564655s) [upload=12188]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000728925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
16 out of 60 is failed (threshold=3) [..........................................X..XXXXXXXXXXXXXXX] duration:
30.000886332s
FAIL (2025-08-07 21:24:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000902069s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration:
50.940603ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:24:37 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 50.988911ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration:
30.001290975s
FAIL (2025-08-07 21:24:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001310033s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...............................................X..XXXXXXXXXX] duration:
30.001094825s
FAIL (2025-08-07 21:24:26 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments...
Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload
Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload
Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments...
Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false
Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded
Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments...
Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded
Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload
Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload
Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments...
Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false
Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded
Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments...
Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded
Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload
Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload
Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments...
Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false
Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded
Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments...
Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded
Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload
Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload
Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments...
Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false
Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded
Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments...
Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded
Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload
Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload
Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments...
Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false
Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded
Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments...
Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded
Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload
Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload
Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments...
Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false
Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded
Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments...
Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded
Aug 7 21:23:56.987167215 (11h44m31.502269791s) [upload=12191]: Starting upload
Aug 7 21:23:56.987252637 (11h44m31.502355205s) [upload=12191]: Done starting upload
Aug 7 21:23:56.987371700 (11h44m31.502474265s) [upload=12191]: Uploading segments...
Aug 7 21:23:56.987567672 (11h44m31.502670242s) [upload=12191]: Got next segment. Inline: false
Aug 7 21:23:56.987605135 (11h44m31.502707711s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:56.987610979 (11h44m31.502713558s) [upload=12191]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:24:26.952565538 (11h45m1.467668121s) [upload=12191]: Next segment err: context deadline exceeded
Aug 7 21:24:26.952696871 (11h45m1.467799434s) [upload=12191]: Done uploading segments...
Aug 7 21:24:26.952716003 (11h45m1.467818567s) [upload=12191]: Upload finished. err: context deadline exceeded
Duration: 30.001122282s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
13 out of 60 is failed (threshold=3) [..............................................X.XXXXXXXXXXXX] duration:
30.001060423s
FAIL (2025-08-07 21:24:17 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:47.433506573 (11h44m21.948609159s) [upload=12187]: Starting upload Aug 7 21:23:47.433620263 (11h44m21.948722851s) [upload=12187]: Done starting upload Aug 7 21:23:47.433723354 (11h44m21.948825934s) [upload=12187]: Uploading segments... Aug 7 21:23:47.433789359 (11h44m21.948891988s) [upload=12187]: Got next segment. Inline: true Aug 7 21:23:47.433801211 (11h44m21.948903796s) [upload=12187]: Waiting for error group managing segments... Aug 7 21:23:47.433851410 (11h44m21.948953994s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:47.433856752 (11h44m21.948959337s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:47.433860816 (11h44m21.948963402s) [upload=12187]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:24:17.403355522 (11h44m51.918458089s) [upload=12187]: Done uploading segments... Aug 7 21:24:17.403383376 (11h44m51.918485952s) [upload=12187]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001097256s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration:
30.00141857s
FAIL (2025-08-07 21:24:16 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001445235s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
15 out of 60 is failed (threshold=3) [...........................................X..XXXXXXXXXXXXXX] duration:
30.001582075s
FAIL (2025-08-07 21:24:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001601802s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
31.448896ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:24:08 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 31.570487ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
30.000757349s
FAIL (2025-08-07 21:24:03 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000783762s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration:
1m0.000883138s
FAIL (2025-08-07 21:24:02 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000976537s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration:
1m0.000577753s
FAIL (2025-08-07 21:23:50 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g upload Aug 7 21:13:22.079907596 (11h33m56.595010178s) [upload=12027]: Uploading segments... Aug 7 21:13:22.079966866 (11h33m56.595069445s) [upload=12027]: Got next segment. Inline: true Aug 7 21:13:22.079973866 (11h33m56.595076443s) [upload=12027]: Waiting for error group managing segments... Aug 7 21:13:22.080019901 (11h33m56.595122472s) [upload=12027]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:22.080024370 (11h33m56.595126947s) [upload=12027]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:22.080029157 (11h33m56.595131732s) [upload=12027]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:22.199851686 (11h33m56.714954269s) [upload=12027]: Done uploading segments... Aug 7 21:13:22.199871202 (11h33m56.714973784s) [upload=12027]: Upload finished. err: <nil> Aug 7 21:14:11.619394074 (11h34m46.134496661s) [upload=12040]: Starting upload Aug 7 21:14:11.619503144 (11h34m46.134605739s) [upload=12040]: Done starting upload Aug 7 21:14:11.619622676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:50.295207015 (11h43m24.810309592s) [upload=12175]: Starting upload Aug 7 21:22:50.295271458 (11h43m24.810374034s) [upload=12175]: Done starting upload Aug 7 21:22:50.295340743 (11h43m24.810443311s) [upload=12175]: Uploading segments... Aug 7 21:22:50.295378631 (11h43m24.810481325s) [upload=12175]: Got next segment. Inline: true Aug 7 21:22:50.295383370 (11h43m24.810485946s) [upload=12175]: Waiting for error group managing segments... Aug 7 21:22:50.295423132 (11h43m24.810525698s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:50.295427541 (11h43m24.810530116s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:50.295430882 (11h43m24.810533459s) [upload=12175]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:50.265121420 (11h44m24.780223985s) [upload=12175]: Done uploading segments... Aug 7 21:23:50.265158624 (11h44m24.780261192s) [upload=12175]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.000598583s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
12 out of 60 is failed (threshold=3) [...............................................X.XXXXXXXXXXX] duration:
30.001674541s
FAIL (2025-08-07 21:23:40 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:10.121114910 (11h43m44.63621749s) [upload=12179]: Starting upload Aug 7 21:23:10.121202614 (11h43m44.636305193s) [upload=12179]: Done starting upload Aug 7 21:23:10.121288037 (11h43m44.636390615s) [upload=12179]: Uploading segments... Aug 7 21:23:10.121347364 (11h43m44.636449949s) [upload=12179]: Got next segment. Inline: true Aug 7 21:23:10.121352991 (11h43m44.636455553s) [upload=12179]: Waiting for error group managing segments... Aug 7 21:23:10.121405392 (11h43m44.636507959s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:23:10.121411077 (11h43m44.636513644s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:10.121414820 (11h43m44.636517397s) [upload=12179]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:23:40.092899455 (11h44m14.608002031s) [upload=12179]: Done uploading segments... Aug 7 21:23:40.092924777 (11h44m14.60802736s) [upload=12179]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001703239s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [...................................................XXX......] duration:
47.403092ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:23:39 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 47.427543ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration:
30.000941326s
FAIL (2025-08-07 21:23:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000956608s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
14 out of 60 is failed (threshold=3) [............................................X..XXXXXXXXXXXXX] duration:
30.001116816s
FAIL (2025-08-07 21:23:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001157488s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [................................................X..XXXXXXXXX] duration:
30.002091641s
FAIL (2025-08-07 21:23:30 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
95929s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.182082323 (11h35m39.697184889s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.182627665 (11h35m39.697730245s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.183210775 (11h35m39.698313344s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.183776547 (11h35m39.69887913s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments...
Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload
Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload
Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments...
Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false
Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded
Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments...
Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded
Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload
Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload
Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments...
Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false
Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded
Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments...
Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded
Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload
Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload
Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments...
Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false
Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded
Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments...
Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded
Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload
Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload
Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments...
Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false
Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded
Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments...
Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded
Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload
Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload
Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments...
Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false
Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded
Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments...
Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded
Aug 7 21:23:00.512106383 (11h43m35.027208949s) [upload=12177]: Starting upload
Aug 7 21:23:00.512179768 (11h43m35.027282345s) [upload=12177]: Done starting upload
Aug 7 21:23:00.512296321 (11h43m35.027398898s) [upload=12177]: Uploading segments...
Aug 7 21:23:00.512613087 (11h43m35.027715655s) [upload=12177]: Got next segment. Inline: false
Aug 7 21:23:00.512683706 (11h43m35.027786273s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:23:00.512689127 (11h43m35.027791703s) [upload=12177]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:23:30.477491924 (11h44m4.992594503s) [upload=12177]: Next segment err: context deadline exceeded
Aug 7 21:23:30.477645840 (11h44m4.992748409s) [upload=12177]: Done uploading segments...
Aug 7 21:23:30.477663897 (11h44m4.992766468s) [upload=12177]: Upload finished. err: context deadline exceeded
Duration: 30.002111703s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration:
30.001312409s
FAIL (2025-08-07 21:23:16 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001346838s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration:
30.001009838s
FAIL (2025-08-07 21:23:09 UTC)
operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.001022993s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
151.62285ms remote:
"tcp:136.0.77.2:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222
FAIL (2025-08-07 21:23:02 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 151.643609ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration:
30.001345359s
FAIL (2025-08-07 21:23:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001369293s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.................................................X..XXXXXXXX] duration:
30.000469021s
FAIL (2025-08-07 21:22:57 UTC)
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
load=12056]: Uploading piece...
Aug 7 21:15:05.043348608 (11h35m39.558451196s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Uploading piece...
Aug 7 21:15:05.043913089 (11h35m39.559015677s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Uploading piece...
Aug 7 21:15:05.051646181 (11h35m39.566748771s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Uploading piece...
Aug 7 21:15:05.177029777 (11h35m39.692132351s) [upload=12056]: Next returned nil segment
Aug 7 21:15:05.177048224 (11h35m39.692150808s) [upload=12056]: Waiting for error group managing segments...
Aug 7 21:15:05.181493366 (11h35m39.696595929s) [noise=true , address=45.91.116.12:4226 , piece_id=3N3F7RJEVNYLV7EFRP3W3X5Z6PLNCZDH6XT5UQVDRV6IYXWCGANA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.182082323 (11h35m39.697184889s) [noise=true , address=45.91.116.11:4119 , piece_id=NAGZ7C7XYS7HEH24GD6CQMRUULZMG5GMVGNJ5L5CVXONOZ4Y36CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.182627665 (11h35m39.697730245s) [noise=true , address=77.90.31.19:4842 , piece_id=IKYGBTCOYPSC4R4HRCW4YPMQUC2R4OVBFAD65UEFJSB2BBD6MO5Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.183210775 (11h35m39.698313344s) [noise=true , address=45.91.116.7:4011 , piece_id=PLTVR5ME4TKVCCFEHYIQYXOJ7JSJLEGBKR2VJT466KIYNAMZPS7Q, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.183776547 (11h35m39.69887913s) [noise=true , address=45.91.116.5:4298 , piece_id=XLGVZ3Y6CPSMHRPCOEO2GTU6HVHTSRALOVZPDYSXDSIECXIXJ7OA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.184100393 (11h35m39.699202963s) [noise=true , address=45.91.116.4:4939 , piece_id=UM2WOS42W7QGQW36MCWQGTUQS44CWS7SBYIVJQFTOHHGRNXILDZQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.184388388 (11h35m39.699490959s) [noise=true , address=45.91.116.6:4974 , piece_id=5VGBFMSP3LAGD46ZIXXXNWJAE2DR5QDTBZBCXDIHXHL2CG3OBLQQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.185120907 (11h35m39.700223474s) [noise=true , address=77.90.31.18:4805 , piece_id=FHWCWZY3AJ53PI576K7IFQJWU3AORQYRCCQXVX7HHUDYU5XJM5IA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.185748114 (11h35m39.700850683s) [noise=true , address=104.36.82.8:4969 , piece_id=IM7246GVGHCFJZ4GVVLA6JOQXOUE7JZSGLMKYW5BHCNEZE27ORPQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186375273 (11h35m39.701477841s) [noise=true , address=104.36.82.8:4971 , piece_id=OVXYQEECN4EAJ22CLCCPPFKSAVB34CROFNS645IGNVKVHRTP35ZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.186905671 (11h35m39.702008238s) [noise=true , address=104.36.82.6:4875 , piece_id=JGONSCWN7IOF4WHT2W7C2FCOC3PU4HIZS7IFD4D5U4OP3NRNIVNA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187080351 (11h35m39.702182922s) [noise=true , address=45.91.116.9:4261 , piece_id=ATQ5QYTM65Q6V2I3WXI44UBCXJCPTLQ3YWTEP574HTCC5FNZDBZA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187463849 (11h35m39.702566421s) [noise=true , address=104.36.82.11:4114 , piece_id=6IVHXGQINEZCTU5VCESQNAOIZNDSOZTCJRECX3RZUOBPYQ2IB5CA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: <nil>
Aug 7 21:15:05.187500876 (11h35m39.702603445s) [seg_pos={0 0}, upload=12056]: Segment reached optimal threshold of 13 pieces.
Aug 7 21:15:05.187666737 (11h35m39.702769308s) [noise=true , address=104.36.82.5:4839 , piece_id=ZQRM42A6G754EAQPPIVJEHUSDHYFNYWFZIY4NSFMMKP5NT3KWOIQ, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187699901 (11h35m39.702802472s) [noise=true , address=104.36.82.2:4731 , piece_id=QZWI4NPCFTAZ5EZK2K6U6NLJLHJ4B5KGOLA23JWKRYMOQSGFDVEA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187755770 (11h35m39.702858351s) [noise=true , address=104.36.82.4:4804 , piece_id=A25BHLKJDHSMMY7BM6KEUXEQBFNYB47BMCX6XCP3OJQP2MAEWKAA, seg_pos={0 0}, upload=12056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF
Aug 7 21:15:05.187785150 (11h35m39.702887738s) [seg_pos={0 0}, upload=12056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 21:15:05.187865579 (11h35m39.70296817s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 21:15:05.187872407 (11h35m39.702974997s) [upload=12056]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 21:19:12.162955387 (11h39m46.678057965s) [upload=12056]: Done uploading segments...
Aug 7 21:19:12.162998431 (11h39m46.678101011s) [upload=12056]: Upload finished. err: metaclient: context deadline exceeded
Aug 7 21:19:12.271592802 (11h39m46.786695387s) [upload=12116]: Starting upload
Aug 7 21:19:12.271672035 (11h39m46.786774611s) [upload=12116]: Done starting upload
Aug 7 21:19:12.271852742 (11h39m46.78695532s) [upload=12116]: Uploading segments...
Aug 7 21:19:12.272263292 (11h39m46.787365868s) [upload=12116]: Got next segment. Inline: false
Aug 7 21:19:12.272444150 (11h39m46.787546726s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:12.272450476 (11h39m46.787553053s) [upload=12116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:19:42.235571988 (11h40m16.75067457s) [upload=12116]: Next segment err: context deadline exceeded
Aug 7 21:19:42.235742671 (11h40m16.750845238s) [upload=12116]: Done uploading segments...
Aug 7 21:19:42.235763358 (11h40m16.750865922s) [upload=12116]: Upload finished. err: context deadline exceeded
Aug 7 21:19:42.346509215 (11h40m16.861611783s) [upload=12126]: Starting upload
Aug 7 21:19:42.346631944 (11h40m16.861734524s) [upload=12126]: Done starting upload
Aug 7 21:19:42.346734761 (11h40m16.86183733s) [upload=12126]: Uploading segments...
Aug 7 21:19:42.347098388 (11h40m16.862200958s) [upload=12126]: Got next segment. Inline: false
Aug 7 21:19:42.347151783 (11h40m16.862254359s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:19:42.347156772 (11h40m16.862259349s) [upload=12126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:20:12.312649136 (11h40m46.827751721s) [upload=12126]: Next segment err: context deadline exceeded
Aug 7 21:20:12.312726339 (11h40m46.827828917s) [upload=12126]: Done uploading segments...
Aug 7 21:20:12.312750219 (11h40m46.827852799s) [upload=12126]: Upload finished. err: context deadline exceeded
Aug 7 21:20:30.404712299 (11h41m4.919814883s) [upload=12142]: Starting upload
Aug 7 21:20:30.404819460 (11h41m4.919922046s) [upload=12142]: Done starting upload
Aug 7 21:20:30.405017708 (11h41m4.920120286s) [upload=12142]: Uploading segments...
Aug 7 21:20:30.405300351 (11h41m4.920402913s) [upload=12142]: Got next segment. Inline: false
Aug 7 21:20:30.405363494 (11h41m4.920466074s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:20:30.405369188 (11h41m4.920471752s) [upload=12142]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:00.366269663 (11h41m34.881372243s) [upload=12142]: Next segment err: context deadline exceeded
Aug 7 21:21:00.366411581 (11h41m34.881514166s) [upload=12142]: Done uploading segments...
Aug 7 21:21:00.366445407 (11h41m34.881547985s) [upload=12142]: Upload finished. err: context deadline exceeded
Aug 7 21:21:23.631753822 (11h41m58.146856392s) [upload=12153]: Starting upload
Aug 7 21:21:23.631820376 (11h41m58.146922952s) [upload=12153]: Done starting upload
Aug 7 21:21:23.631936735 (11h41m58.147039303s) [upload=12153]: Uploading segments...
Aug 7 21:21:23.632090665 (11h41m58.147193229s) [upload=12153]: Got next segment. Inline: false
Aug 7 21:21:23.632132835 (11h41m58.147235423s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:21:23.632138285 (11h41m58.147240868s) [upload=12153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:21:53.583630847 (11h42m28.098733413s) [upload=12153]: Next segment err: context deadline exceeded
Aug 7 21:21:53.583727819 (11h42m28.0988304s) [upload=12153]: Done uploading segments...
Aug 7 21:21:53.583776993 (11h42m28.098879569s) [upload=12153]: Upload finished. err: context deadline exceeded
Aug 7 21:22:27.411174697 (11h43m1.926277275s) [upload=12166]: Starting upload
Aug 7 21:22:27.411234640 (11h43m1.926337216s) [upload=12166]: Done starting upload
Aug 7 21:22:27.411365574 (11h43m1.926468153s) [upload=12166]: Uploading segments...
Aug 7 21:22:27.411683232 (11h43m1.926785815s) [upload=12166]: Got next segment. Inline: false
Aug 7 21:22:27.411802539 (11h43m1.926905127s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 21:22:27.411808914 (11h43m1.926911491s) [upload=12166]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 21:22:57.370868197 (11h43m31.885970767s) [upload=12166]: Next segment err: context deadline exceeded
Aug 7 21:22:57.371007492 (11h43m31.886110059s) [upload=12166]: Done uploading segments...
Aug 7 21:22:57.371048788 (11h43m31.886151358s) [upload=12166]: Upload finished. err: context deadline exceeded
Duration: 30.000490698s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration:
1m0.000604802s
FAIL (2025-08-07 21:22:54 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000643146s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration:
220.369598ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:22:52 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 220.399399ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
13 out of 60 is failed (threshold=3) [.............................................X..XXXXXXXXXXXX] duration:
30.001353214s
FAIL (2025-08-07 21:22:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001366407s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration:
1m0.001717007s
FAIL (2025-08-07 21:22:42 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1007s) [upload=12014]: Done starting upload Aug 7 21:12:44.577778064 (11h33m19.09288064s) [upload=12014]: Uploading segments... Aug 7 21:12:44.577870249 (11h33m19.09297311s) [upload=12014]: Got next segment. Inline: true Aug 7 21:12:44.577878347 (11h33m19.09298093s) [upload=12014]: Waiting for error group managing segments... Aug 7 21:12:44.577949813 (11h33m19.093052392s) [upload=12014]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:12:44.577956047 (11h33m19.093058626s) [upload=12014]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:12:44.577963216 (11h33m19.093065803s) [upload=12014]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:12:44.741155623 (11h33m19.256258199s) [upload=12014]: Done uploading segments... Aug 7 21:12:44.741173325 (11h33m19.256275891s) [upload=12014]: Upload finished. err: <nil> Aug 7 21:13:22.079763163 (11h33m56.594865742s) [upload=12027]: Starting upload Aug 7 21:13:22.079827372 (11h33m56.594929946s) [upload=12027]: Done starting upload Aug 7 21:13:22.079907596 (11h33m56.595010178s) [upload=12027]: Uploading segments... Aug 7 21:13:22.079966866 (11h33m56.595069445s) [upload=12027]: Got next segment. Inline: true Aug 7 21:13:22.079973866 (11h33m56.595076443s) [upload=12027]: Waiting for error group managing segments... Aug 7 21:13:22.080019901 (11h33m56.595122472s) [upload=12027]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:13:22.080024370 (11h33m56.595126947s) [upload=12027]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:13:22.080029157 (11h33m56.595131732s) [upload=12027]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:13:22.199851686 (11h33m56.714954269s) [upload=12027]: Done uploading segments... Aug 7 21:13:22.199871202 (11h33m56.714973784s) [upload=12027]: Upload finished. err: <nil> Aug 7 21:14:11.619394074 (11h34m46.134496661s) [upload=12040]: Starting upload Aug 7 21:14:11.619503144 (11h34m46.134605739s) [upload=12040]: Done starting upload Aug 7 21:14:11.619622676 (11h34m46.13472524s) [upload=12040]: Uploading segments... Aug 7 21:14:11.619678842 (11h34m46.134781443s) [upload=12040]: Got next segment. Inline: true Aug 7 21:14:11.619698988 (11h34m46.134801579s) [upload=12040]: Waiting for error group managing segments... Aug 7 21:14:11.619745714 (11h34m46.1348483s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:11.619750849 (11h34m46.134853438s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:11.619754200 (11h34m46.134856779s) [upload=12040]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:11.587599715 (11h35m46.102702292s) [upload=12040]: Done uploading segments... Aug 7 21:15:11.587622036 (11h35m46.102724617s) [upload=12040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:11.944029087 (11h35m46.459131673s) [upload=12059]: Starting upload Aug 7 21:15:11.944115611 (11h35m46.459218173s) [upload=12059]: Done starting upload Aug 7 21:15:11.944188828 (11h35m46.459291405s) [upload=12059]: Uploading segments... Aug 7 21:15:11.944299925 (11h35m46.459402511s) [upload=12059]: Got next segment. Inline: true Aug 7 21:15:11.944487606 (11h35m46.459590195s) [upload=12059]: Waiting for error group managing segments... Aug 7 21:15:11.944583177 (11h35m46.459685757s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:11.944589579 (11h35m46.459692164s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:11.944593842 (11h35m46.459696426s) [upload=12059]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:11.913962661 (11h36m46.429065258s) [upload=12059]: Done uploading segments... Aug 7 21:16:11.914008028 (11h36m46.429110614s) [upload=12059]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:23.879062977 (11h36m58.394165965s) [upload=12076]: Starting upload Aug 7 21:16:23.879174498 (11h36m58.394277072s) [upload=12076]: Done starting upload Aug 7 21:16:23.879271835 (11h36m58.394374463s) [upload=12076]: Uploading segments... Aug 7 21:16:23.879342820 (11h36m58.3944454s) [upload=12076]: Got next segment. Inline: true Aug 7 21:16:23.879348075 (11h36m58.394450651s) [upload=12076]: Waiting for error group managing segments... Aug 7 21:16:23.879396205 (11h36m58.394498782s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:23.879404606 (11h36m58.394507185s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:23.879411048 (11h36m58.394513625s) [upload=12076]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:23.849739084 (11h37m58.364841665s) [upload=12076]: Done uploading segments... Aug 7 21:17:23.849762662 (11h37m58.364865249s) [upload=12076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:31.003058721 (11h38m5.518161287s) [upload=12091]: Starting upload Aug 7 21:17:31.003131817 (11h38m5.518234394s) [upload=12091]: Done starting upload Aug 7 21:17:31.003209144 (11h38m5.518311711s) [upload=12091]: Uploading segments... Aug 7 21:17:31.003304802 (11h38m5.51840739s) [upload=12091]: Got next segment. Inline: true Aug 7 21:17:31.003312999 (11h38m5.518415601s) [upload=12091]: Waiting for error group managing segments... Aug 7 21:17:31.003391570 (11h38m5.51849415s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:31.003399048 (11h38m5.518501628s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:31.003403157 (11h38m5.518505738s) [upload=12091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:30.975282177 (11h39m5.490384768s) [upload=12091]: Done uploading segments... Aug 7 21:18:30.975306141 (11h39m5.490408719s) [upload=12091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:31.084603523 (11h39m5.599706108s) [upload=12106]: Starting upload Aug 7 21:18:31.084707053 (11h39m5.599809631s) [upload=12106]: Done starting upload Aug 7 21:18:31.084838746 (11h39m5.59994131s) [upload=12106]: Uploading segments... Aug 7 21:18:31.084971632 (11h39m5.60007421s) [upload=12106]: Got next segment. Inline: true Aug 7 21:18:31.084978232 (11h39m5.60008081s) [upload=12106]: Waiting for error group managing segments... Aug 7 21:18:31.085029173 (11h39m5.600131751s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:31.085045347 (11h39m5.600147925s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:31.085049038 (11h39m5.600151616s) [upload=12106]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:31.052455547 (11h40m5.567558132s) [upload=12106]: Done uploading segments... Aug 7 21:19:31.052478807 (11h40m5.567581383s) [upload=12106]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:36.170902647 (11h40m10.686005227s) [upload=12124]: Starting upload Aug 7 21:19:36.170975299 (11h40m10.686077875s) [upload=12124]: Done starting upload Aug 7 21:19:36.171070498 (11h40m10.686173066s) [upload=12124]: Uploading segments... Aug 7 21:19:36.171157004 (11h40m10.686259589s) [upload=12124]: Got next segment. Inline: true Aug 7 21:19:36.171163478 (11h40m10.686266055s) [upload=12124]: Waiting for error group managing segments... Aug 7 21:19:36.171226869 (11h40m10.686329445s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:36.171232162 (11h40m10.686334738s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:36.171235498 (11h40m10.686338074s) [upload=12124]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:36.141416725 (11h41m10.656519301s) [upload=12124]: Done uploading segments... Aug 7 21:20:36.141434181 (11h41m10.656536758s) [upload=12124]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.103436676 (11h41m16.618539247s) [upload=12143]: Starting upload Aug 7 21:20:42.103499312 (11h41m16.618601886s) [upload=12143]: Done starting upload Aug 7 21:20:42.103560058 (11h41m16.618662626s) [upload=12143]: Uploading segments... Aug 7 21:20:42.103602942 (11h41m16.61870553s) [upload=12143]: Got next segment. Inline: true Aug 7 21:20:42.103608733 (11h41m16.618711318s) [upload=12143]: Waiting for error group managing segments... Aug 7 21:20:42.103655574 (11h41m16.618758155s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.103660237 (11h41m16.618762817s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.103663636 (11h41m16.618766219s) [upload=12143]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:42.072486318 (11h42m16.587588892s) [upload=12143]: Done uploading segments... Aug 7 21:21:42.072511657 (11h42m16.587614233s) [upload=12143]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:42.187716607 (11h42m16.702819176s) [upload=12157]: Starting upload Aug 7 21:21:42.187836773 (11h42m16.702939361s) [upload=12157]: Done starting upload Aug 7 21:21:42.187975832 (11h42m16.70307841s) [upload=12157]: Uploading segments... Aug 7 21:21:42.188045957 (11h42m16.703148542s) [upload=12157]: Got next segment. Inline: true Aug 7 21:21:42.188051805 (11h42m16.703154382s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:21:42.188097961 (11h42m16.703200526s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:42.188105028 (11h42m16.703207604s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:42.188108433 (11h42m16.703211009s) [upload=12157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:42.156253831 (11h43m16.671356412s) [upload=12157]: Done uploading segments... Aug 7 21:22:42.156277094 (11h43m16.671379673s) [upload=12157]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001732279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration:
2m21.15051103s
FAIL (2025-08-07 21:22:39 UTC)
operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2m21.150525843s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
11 out of 60 is failed (threshold=3) [................................................X.XXXXXXXXXX] duration:
30.001275222s
FAIL (2025-08-07 21:22:30 UTC)
operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9.910167846s) [upload=12049]: Uploading segments... Aug 7 21:14:45.395104752 (11h35m19.910207337s) [upload=12049]: Got next segment. Inline: true Aug 7 21:14:45.395110169 (11h35m19.910212745s) [upload=12049]: Waiting for error group managing segments... Aug 7 21:14:45.395152641 (11h35m19.910255207s) [upload=12049]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.395157021 (11h35m19.910259598s) [upload=12049]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.395161401 (11h35m19.910263982s) [upload=12049]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.359522906 (11h35m49.874625488s) [upload=12049]: Done uploading segments... Aug 7 21:15:15.359549039 (11h35m49.874651622s) [upload=12049]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:15:53.553631580 (11h36m28.068734159s) [upload=12066]: Starting upload Aug 7 21:15:53.553693025 (11h36m28.068795593s) [upload=12066]: Done starting upload Aug 7 21:15:53.553757062 (11h36m28.06885963s) [upload=12066]: Uploading segments... Aug 7 21:15:53.553809335 (11h36m28.068911922s) [upload=12066]: Got next segment. Inline: true Aug 7 21:15:53.553813895 (11h36m28.068916471s) [upload=12066]: Waiting for error group managing segments... Aug 7 21:15:53.553863687 (11h36m28.068966263s) [upload=12066]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:15:53.553867873 (11h36m28.068970453s) [upload=12066]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:53.553871084 (11h36m28.068973661s) [upload=12066]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:16:23.520202496 (11h36m58.035305071s) [upload=12066]: Done uploading segments... Aug 7 21:16:23.520223689 (11h36m58.035326265s) [upload=12066]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:49.902511179 (11h37m24.417613746s) [upload=12081]: Starting upload Aug 7 21:16:49.902589681 (11h37m24.417692258s) [upload=12081]: Done starting upload Aug 7 21:16:49.902694562 (11h37m24.41779713s) [upload=12081]: Uploading segments... Aug 7 21:16:49.902734709 (11h37m24.417837294s) [upload=12081]: Got next segment. Inline: true Aug 7 21:16:49.902739795 (11h37m24.417842425s) [upload=12081]: Waiting for error group managing segments... Aug 7 21:16:49.902793344 (11h37m24.41789592s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:16:49.902798176 (11h37m24.417900742s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:49.902801774 (11h37m24.417904341s) [upload=12081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:17:19.864701484 (11h37m54.379804061s) [upload=12081]: Done uploading segments... Aug 7 21:17:19.864724958 (11h37m54.379827534s) [upload=12081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:42.355620767 (11h38m16.870723344s) [upload=12093]: Starting upload Aug 7 21:17:42.355687087 (11h38m16.870789667s) [upload=12093]: Done starting upload Aug 7 21:17:42.355755667 (11h38m16.870858244s) [upload=12093]: Uploading segments... Aug 7 21:17:42.355797553 (11h38m16.870900136s) [upload=12093]: Got next segment. Inline: true Aug 7 21:17:42.355802471 (11h38m16.870905047s) [upload=12093]: Waiting for error group managing segments... Aug 7 21:17:42.355844822 (11h38m16.870947397s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:17:42.355849310 (11h38m16.870951886s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:42.355852551 (11h38m16.870955127s) [upload=12093]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:12.325124462 (11h38m46.840227039s) [upload=12093]: Done uploading segments... Aug 7 21:18:12.325149807 (11h38m46.840252384s) [upload=12093]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:12.438160910 (11h38m46.953263476s) [upload=12101]: Starting upload Aug 7 21:18:12.438234374 (11h38m46.953336964s) [upload=12101]: Done starting upload Aug 7 21:18:12.438310223 (11h38m46.953412791s) [upload=12101]: Uploading segments... Aug 7 21:18:12.438356680 (11h38m46.953459262s) [upload=12101]: Got next segment. Inline: true Aug 7 21:18:12.438362121 (11h38m46.953464698s) [upload=12101]: Waiting for error group managing segments... Aug 7 21:18:12.438410226 (11h38m46.95351281s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:18:12.438415407 (11h38m46.953517988s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:12.438419222 (11h38m46.953521801s) [upload=12101]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:18:42.409203896 (11h39m16.924306477s) [upload=12101]: Done uploading segments... Aug 7 21:18:42.409255203 (11h39m16.924357785s) [upload=12101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:21.467083582 (11h39m55.982186149s) [upload=12118]: Starting upload Aug 7 21:19:21.467170698 (11h39m55.982273276s) [upload=12118]: Done starting upload Aug 7 21:19:21.467252181 (11h39m55.982354752s) [upload=12118]: Uploading segments... Aug 7 21:19:21.467328304 (11h39m55.982430889s) [upload=12118]: Got next segment. Inline: true Aug 7 21:19:21.467334874 (11h39m55.982437451s) [upload=12118]: Waiting for error group managing segments... Aug 7 21:19:21.467410316 (11h39m55.982512883s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:21.467416752 (11h39m55.982519342s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:21.467423866 (11h39m55.982526451s) [upload=12118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:19:51.434574480 (11h40m25.949677057s) [upload=12118]: Done uploading segments... Aug 7 21:19:51.434595758 (11h40m25.949698333s) [upload=12118]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:55.049961158 (11h40m29.565063742s) [upload=12130]: Starting upload Aug 7 21:19:55.050049645 (11h40m29.565152234s) [upload=12130]: Done starting upload Aug 7 21:19:55.050174766 (11h40m29.56527734s) [upload=12130]: Uploading segments... Aug 7 21:19:55.050230680 (11h40m29.565333265s) [upload=12130]: Got next segment. Inline: true Aug 7 21:19:55.050236346 (11h40m29.565338922s) [upload=12130]: Waiting for error group managing segments... Aug 7 21:19:55.050278491 (11h40m29.565381058s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:19:55.050282931 (11h40m29.56538551s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:55.050297706 (11h40m29.565400282s) [upload=12130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:20:25.015067991 (11h40m59.530170581s) [upload=12130]: Done uploading segments... Aug 7 21:20:25.015093812 (11h40m59.530196392s) [upload=12130]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:42.115589558 (11h41m16.630692139s) [upload=12144]: Starting upload Aug 7 21:20:42.115693469 (11h41m16.63079605s) [upload=12144]: Done starting upload Aug 7 21:20:42.115777911 (11h41m16.630880476s) [upload=12144]: Uploading segments... Aug 7 21:20:42.115831615 (11h41m16.6309342s) [upload=12144]: Got next segment. Inline: true Aug 7 21:20:42.115837889 (11h41m16.630940468s) [upload=12144]: Waiting for error group managing segments... Aug 7 21:20:42.115891141 (11h41m16.630993706s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:20:42.115895872 (11h41m16.630998437s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:42.115899097 (11h41m16.631001676s) [upload=12144]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:21:12.084618276 (11h41m46.599720859s) [upload=12144]: Done uploading segments... Aug 7 21:21:12.084644433 (11h41m46.599747012s) [upload=12144]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:30.171137892 (11h42m4.686240458s) [upload=12154]: Starting upload Aug 7 21:21:30.171221513 (11h42m4.686324082s) [upload=12154]: Done starting upload Aug 7 21:21:30.171329274 (11h42m4.686431837s) [upload=12154]: Uploading segments... Aug 7 21:21:30.171405110 (11h42m4.686507697s) [upload=12154]: Got next segment. Inline: true Aug 7 21:21:30.171410301 (11h42m4.686512877s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:21:30.171472311 (11h42m4.68657488s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:21:30.171476980 (11h42m4.686579557s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:30.171480121 (11h42m4.686582687s) [upload=12154]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:00.141348462 (11h42m34.656451029s) [upload=12154]: Done uploading segments... Aug 7 21:22:00.141367880 (11h42m34.656470446s) [upload=12154]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:00.248764209 (11h42m34.763866777s) [upload=12161]: Starting upload Aug 7 21:22:00.248839327 (11h42m34.763941903s) [upload=12161]: Done starting upload Aug 7 21:22:00.248916077 (11h42m34.764018654s) [upload=12161]: Uploading segments... Aug 7 21:22:00.248957746 (11h42m34.764060332s) [upload=12161]: Got next segment. Inline: true Aug 7 21:22:00.248974248 (11h42m34.764076825s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:22:00.249015723 (11h42m34.76411829s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 21:22:00.249020198 (11h42m34.764122774s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:00.249023336 (11h42m34.764125913s) [upload=12161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 21:22:30.219103896 (11h43m4.734206496s) [upload=12161]: Done uploading segments... Aug 7 21:22:30.219144601 (11h43m4.734247178s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded
Duration: 30.001295654s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration:
53.764168ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-07 21:22:22 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 53.796649ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
12 out of 60 is failed (threshold=3) [..............................................X..XXXXXXXXXXX] duration:
30.000801249s
FAIL (2025-08-07 21:22:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00087167s
edge
gateway
us-select-1
|